Mr Mori said Japan was "not considering cancelling the Olympics at all", saying plans for the start of the Olympic torch relay on 26 March would go ahead.
Organisers will monitor the coronavirus pandemic for four weeks before making a decision on how to proceed, he said, with postponement among a number of options.
His comments came after Canada said it was pulling its athletes out of the games and urged the International OIympic Committee (IOC) "to postpone the games for one year".
Japanese prime minister Shinzo Abe, speaking to parliament on Monday, said postponing the Olympics was an option if holding them in their "complete form" became impossible.
"If that becomes difficult, we may have no option but to consider postponing the games, given the Olympic principle of putting the health of athletes first."
Speaking at a news conference, Mr Mori said he was not "so stupid as to assume athletes and spectators will come to the Olympics amid the global coronavirus crisis, even if we push forward with it".
But citing the amount of money already invested in the games, he said the IOC thought cancelling outright would be "bad for [both] athletes and stakeholders".
Mr Mori said he could not rule out the games being postponed to 2021, but that "as the Tokyo Olympic Committee, we want to hold the Olympics in 2020".
On Sunday, the IOC - whose president Thomas Bach insisted as recently as last week that there was "no plan B" for the games - issued a statement finally accepting that postponement was a possibility.
After a meeting of its executive board, the IOC statement said the time had come for the body "to take the next step in its scenario planning".
"Human lives take precedence over everything, including the staging of the Games,” Mr Bach said. “The IOC wants to be part of the solution. Therefore we have made it our leading principle to safeguard the health of everyone involved."
The Olympic torch arrived on Friday in northern Japan, and thousands crowded to see it in one city despite organisers urging restraint. The torch relay across the nation is still set to begin on Thursday, Mr Mori said, but local organisers in the north may yet decide to suspend it.