Mr Grayling's decision to award Seaborne Freight a contract worth £13.8 million had attracted widespread criticism.
The Department for Transport said it had decided to terminate the contract after the company's Irish backer, Arklow Shipping, pulled out of the deal.
A DfT spokeswoman said: "Following the decision of Seaborne Freight's backer, Arklow Shipping, to step back from the deal, it became clear Seaborne would not reach its contractual requirements with the Government. We have therefore decided to terminate our agreement.
"The Government is already in advanced talks with a number of companies to secure additional freight capacity - including through the port of Ramsgate - in the event of a no-deal Brexit."
Labour seized on the situation to say Mr Grayling should quit or be sacked.
Shadow transport secretary Andy McDonald said: "As we predicted, the Seaborne Freight contract has been cancelled.
"This cannot go without consequence. The Chris Grayling catalogue of calamities grows bigger by the day.
Chris Grayling was told & better told that this wouldn’t work. This is humiliating and causes yet more damage to our country’s reputation. He’s been banned from Calais & Theresa May should ban him from the Dept of Transport before he does any further harm. https://t.co/ue4IuD14i7— Andy McDonald MP (@AndyMcDonaldMP)February 9, 2019
"This contract was never going to work but this Secretary of State, true to form, blunders from one disaster to another.
"Whilst Theresa May needs the few friends she has right now, we cannot have this incompetent Transport Secretary carry on heaping humiliation after humiliation on our country. He has to go."
Labour leader Jeremy Corbyn raised the "ludicrous situation" in a speech in Coventry, saying: "Chris Grayling the Transport Secretary claimed the Government had 'looked very carefully' at Seaborne Freight before giving the company the contract, but apparently not carefully enough to notice that it didn't have any ships.
"Chris Grayling does have form in other departments."
Brexit-backing Tory MP Jacob Rees-Mogg questioned whether Leo Varadkar's Irish Government had any influence on Arklow's decision.
He told the Daily Telegraph: "One has to hope that the Irish Government has not leant on or put any pressure on Arklow to persuade it to pull out.
"That would be a very unfriendly act of a neighbour to obstruct no-deal preparations and one has to hope very sincerely that this is genuinely a corporate decision."
Mr Grayling last month defended the Seaborne Freight contract, insisting it was "not a risk".
It was one of three firms awarded contracts totalling £108 million in late December to lay on additional crossings to ease the pressure on Dover when Britain leaves the EU, despite having never run a Channel service.
Seaborne Freight was due to run a service between Ramsgate and Ostend in the event that Britain leaves the EU with no deal on March 29.
The department said it had been Arklow Shipping's backing that gave it confidence in the viability of the deal, and that it stands by the robust due diligence carried out on Seaborne Freight.
It added no taxpayer money had been transferred to the company.
Mick Cash, general secretary of the Rail, Maritime and Transport (RMT) union, said: "RMT has taken a number of protests over the fiasco of the Government's Brexit ferry contracts to both the Department for Transport and the ports, and the news this morning comes as no surprise to us.
"The whole exercise is a complete and utter shambles with the Government ignoring union calls on what needs to happen. Instead they are blundering on from crisis to crisis.
"RMT has set out a package of demands that would guarantee that the Brexit ferry contracts are crewed by British seafarers, on decent pay and conditions negotiated through recognised trade unions.
"This Government 'wing and a prayer' approach was always doomed to failure and it's time for Chris Grayling to stop attacking RMT and start listening to people who actually know what they are talking about instead of the chancers selling him a pile of old rope they don't even own."