Artificial intelligence (AI) models that control robots are improving faster than the industry’s most optimistic forecasts, as chatbot-style techniques move onto factory floors.
Training data, once a bottleneck for robotics, is getting cheaper as startups produce it in bulk, while generative AI can now build realistic simulations of real-world scenes. Together, these trends are pushing the most valuable learning out of laboratories and onto live factory floors.
“The robot brains are getting smarter incredibly fast, faster than even the most optimistic people expected. It is shocking to us,” said Russ Tedrake, co-founder and chief executive of Walden Robotics and a professor at the Massachusetts Institute of Technology (MIT).
“What we’re doing now in the robotics community is we’re teaching these models one more output modality, which is how do you control your end effector. These models are super powerful, and it doesn’t take that much to uptrain them to be good robot models,” he said.
Generative pre-trained transformer (GPT) models have already moved from text to images, speech and video. The end effector, the gripper or hand at the end of a robot arm, is the next output they'll learn.
Before founding Walden, his team at Toyota Research Institute (TRI) built large simulation and experimental pipelines to measure the scaling laws of physical AI models with rigorous statistics. He said the results looked incredibly promising and are still rising with no end in sight.
“It worked for language models. It worked for computer vision, but robots are different. The real world is harder,” he said, recalling how many roboticists reacted as generative AI took off. “Sorry, robots are not different.”
General “mid-training” data can now be bought off the shelf at falling cost. Data gathered in real applications still matters, much as an intelligent human needs on-the-job training to become an expert.
Stronger base models and cheaper general data are reshaping the “robot data pyramid,” he said.
“Simulation has always played a strong role. But even this month, it’s getting better and better because of the GPT-type models that are so good now at scene generation. We can do real to sim in ways that we just couldn’t before,” Tedrake said.
Walden Robotics, a Cambridge, Massachusetts-based robotics startup, spun out of TRI in January. In July, it emerged from stealth with $300 million in seed funding at a $1.1 billion valuation, in a round co-led by Toyota and Deviation Capital.
Humans finish the job
Tedrake’s case study opened the Physical AI and Robotics Summit, which Economist Enterprise organized in London on October 13. Kenneth Cukier, deputy executive editor of The Economist, hosted the session on machines with minds of their own.
Walden deploys its robots a little differently from some rivals, backing them with remote assistants who step in when the AI falls short.
“If the physical AI is capable of doing the job 98% of the time, that’s okay. We’ll have a remote assistant ready to finish the job and do useful work for the customer,” Tedrake said.
He compared the approach to Waymo, which last reported having about 70 remote operators for a fleet of 3,000 robotaxis. Waymo is an autonomous driving and robotics company owned by Alphabet Inc.
In one factory video, Walden’s AI was about to pick a deeply buried camshaft when a remote assistant intervened and pointed it to an exposed one nearby. The operator made a minor adjustment, then switched the robot back to autonomy and let it continue.
“That’s the way it works. Even when the AI is not perfect, the robot can get the work done,” Tedrake said.
The approach is rooted in Jidoka, one of the three pillars of the Toyota Production System. It dates back about 100 years to automated looms, where a runtime monitor that detected mistakes let one worker oversee 20 machines instead of one.
“We’re doing exactly the same thing with our robots. I’ve done lab experiments long enough. All of the interesting learning and data is in the field,” he said.
Remote help lets Walden shift its data collection to customer sites. The startup was deployed in Toyota factories in the US within three months of its founding, handling:
machine tending
tool setting
kitting
assembly
Its wheeled humanoid can fit a transfer case on its own and seat a bearing in its race.
“It’s pretty hard for humans to do this. We have our autonomy doing this very often, 100% autonomous,” Tedrake said.
Tedrake sees three groups of players in the market:
frontier labs building physical AI models, a list that now includes OpenAI and Anthropic
specialists in hardware and low-level control, such as Boston Dynamics
scrappy startups laser-focused on customers
Some frontier labs also partner with "neo-integrators," new robot integrators that deploy the labs' models for customers.
“Walden is, for better or for worse, putting itself right in the heart of that and trying to do all three,” he said.
Walden’s strategic partners include Samsung, Boeing, CoreWeave and Nvidia, and the startup is up-training strong base models into world-class physical AI systems while expanding its data capability.
“My Slack today was new videos. People going, ‘I cannot believe the robot does that,’” he added.
Pilots are not ROI
The industry’s narrative is shifting from demos to deployment, though deploying robots remains hard. Tedrake said the conversation now needs to move beyond deployments to real customer value.
“A pilot is one thing. ROI (return on investment) is a different thing,” he said.
He said labor replacement and cost reduction can justify robots, and this case is easier to make in the US than in some other places. Yet that view misses the bigger picture of how robots will change factories and the wider world.
He named the company after Henry David Thoreau’s book Walden to keep that bigger picture front and center.
“These robots are going to change not just the future of manufacturing, but the future of physical work, and I want to get that right as much as possible,” he said.
On hardware, Walden builds wheeled humanoids rather than fixed arms. Tedrake said many factory robots sit idle waiting for work, while human workers often handle several jobs within a single cycle time on the production line.
“If we can make the mobile case, then why would you want a robot bolted to the ground?” he said.
Walden also uses simple hands because they must survive full shifts of rough factory work. Even so, its robots can still perform very dexterous tasks.
“I believe completely in dexterous hands. But at the end of the week, the hands look pretty torn up, and we haven’t yet found an off-the-shelf hand that can withstand that beat,” Tedrake said.
His route into physical AI ran through the US Defense Advanced Research Projects Agency (DARPA) Robotics Challenge, where he led MIT’s software team on a Boston Dynamics robot. Closed-door meetings there prompted Google to buy Boston Dynamics and other robotics companies, and Toyota to open labs near MIT and Stanford.
“The way you teach a new task to these robots is not by writing code. It’s no-code. It’s just by providing more demonstrations, and you can quickly add a new task,” he said.
A person wearing a teleoperation interface can show a robot a task 200 to 300 times to train an end-to-end neural network. Even an inexpensive handheld gripper can now provide the demonstrations.
Tedrake said the technology is ready if deployed thoughtfully, but customers have a high bar. Startups that win them over will need a solid business case and long-term plans for manufacturing, deployment, maintenance, policy and insurance in an incredibly competitive market.