The disaster in the popular hot springs resort of Atami unfolded early on Saturday when a torrent of mud crashed down a mountainside following days of heavy rain.
Prime minister Yoshihide Suga said 19 people had been rescued so far after 130 homes and other buildings were damaged. Three people have been injured.
Warning the public to watch out for more landslides as the rainfall continued, he said in a statement that the government was “putting people’s lives above all else.”
Mr Suga added: "I express my heartfelt sympathy to all those who have been affected by the disaster.
“Police, firefighters, and members of the coast guard and self-defense forces are currently going all out to conduct rescue and relief activities and provide guidance to evacuees.”
Residents in the Shizuoka, Kanagawa and Chiba prefectures were ordered to evacuate and more than 2,800 homes in the region were left without power, according to the Tokyo Electric Power Company.
More than 1,000 soldiers, firefighters and police were deployed in the emergency response and six military drones were flown to help assess the area.
Crews were wading through the mud and using search dogs to try and locate survivors among the wreckage.
Mayor Sakae Saito told rescue teams that the "next 72 hours are crucial" at a meeting of the task force on Sunday, according to the Japan Times.
Witnesses have told of hearing a loud roar as the landslide swept through the city.
"I heard a horrible sound and saw a mudslide flowing downwards as rescue workers were urging people to evacuate. So I ran to higher ground," one witness told national broadcaster NHK.
However Mariko Hattori, an interpreter who lives a short walk away from the affected area, said he was initially unaware of the disaster.
"The first things I noticed were lots of emergency vehicles. I didn't know what happened at first," she said. "Then I was frightened when I saw the footage."
Additional reporting by agencies