Shakhtar had to field a weakened team as a result of coronavirus cases, but the Ukrainian underdogs tore Real to shreds as they raced into a 3-0 lead in an incredible first half.
Modric and Vinicius Junior pulled goals back after the break but they were unable to avoid a humiliating loss.
"In the first half we were not worthy of this competition," said Modric.
"It's obvious we need to improve, we lacked confidence today, especially in the first half, but we have to move on, we can't waste any time regretting what has happened."
Real were playing at their 6,000-capacity Alfredo Di Stefano training ground as building work takes place at the Bernabeu, but it is Zinedine Zidane's squad which looks in need of some major renovation.
They were desperate to bounce back from a surprise 1-0 defeat by Cadiz at the weekend, but things quickly went from bad to worse for the 13-time European champions.
Shakhtar gave them a scare after 15 minutes when Marlos burst through one-on-one with Thibaut Courtois.
But the winger's finish was too close to Courtois with the former Chelsea keeper saving with his legs.
However, it was a sign of what was to come and Courtois could do nothing to prevent Tete opening the scoring with a low finish after a strong run from full-back Viktor Korniienko.
Three minutes later the visitors had a second, Courtois parrying Tete's shot only for defender Raphael Varane to prod the rebound into his own net.
Manor Solomon capped a remarkable first half with the third, latching onto Tete's backheel and rolling the ball into the bottom corner.
Zidane threw on Karim Benzema at the break and they pulled one back eight minutes into the second half with a curler from Modric.
Substitute Vinicius Junior then scored within seconds of joining the action with a blistering run and shot.
Shakhtar could have gone further ahead but Courtois made a crucial save to deny Tete before Marlos had a goal ruled out for offside.
Real probed for an equaliser but Benzema was denied by a last-ditch block and Eder Militao fired over.
In stoppage time Federico Valverde thought he had spared Real's blushes with an equaliser but it was ruled out after a video check as Vinicius Junior was offside.
Additional reporting by PA.