Robots are increasingly learning new jobs from a few demonstrations rather than lines of code, letting companies deploy new tasks without reprogramming their machines.
Open-source tools and models are driving the shift, putting robot training within reach of anyone who can record demonstrations.
“The future of robotics won’t be programmed task by task. Robots will learn just by data, and this is fundamentally very different,” said Pepijn Kooijmans, LeRobot lead at Hugging Face.
“Before, I had to program things. You had to chain multiple things together, perception and motion planning,” Kooijmans said. “Now I just did a bunch of demonstrations, 20 minutes, and it suddenly worked without reprogramming anything. This means that the code was moving into data, in just examples.”
He trained in mechatronics before moving into software and artificial intelligence (AI). Google DeepMind’s RT-1 model and videos of Stanford researchers’ pizza-making robot later drew him back to the field. He started experimenting with robot arms and the latest research papers.
He said these systems learn more like humans do. They take in images or video, the robot’s joint angles and a prompt, which can be text or video. An extra component, called an “action expert,” sits on top of pre-trained vision-language models and outputs movements directly, often the next second of motion.
In one setup he showed, cameras at the robot’s base and on each wrist feed a 400-million-parameter vision encoder.
A 2-billion-parameter vision-language model then combines those images with the readings from two seven-joint arms and grippers, plus a typed instruction such as “fold the T-shirt.” A 300-million-parameter action expert turns the result into 30 actions at a time, covering joint angles and gripper commands.
“If you want to deploy a new task, you don’t have to program it again,” he said. “Every improvement that we can make to the data and to the models should and will also work for all the other tasks.”
He said the community and ecosystem around early research were still very small when he entered the field.
Hugging Face, a New York-based AI company best known for hosting the largest online repository of open-source AI models and code, launched LeRobot in Paris in May 2024 to fill that gap.
LeRobot aims to give everyone the learning tools he wished he had when he started. He said it now ranks among the biggest open-source end-to-end robotics libraries, with more than 20,000 people on its Discord server training policies and gathering data. It also works with physics simulators, so developers without robots can test their models.
Open models multiply
The talk took place at the Economist Enterprise Physical AI & Robotics Summit in London on October 13, in a session on how open source is accelerating physical AI. Kenneth Cukier, deputy executive editor of The Economist, hosted the session.
Open-source robot models have multiplied over the past year, and the list keeps growing. Some are pre-trained and can perform tasks immediately, while others require training from scratch on a user’s own data. Kooijmans cited Xiaomi-Robotics-0, Wall-OSS and Nvidia’s Cosmos 3 among recent releases.
“These models have seen a lot of embodiments, a lot of different robots, and they can work without data,” he said. “If you add a bit of data, they can learn a new task much better and more quickly.”
The three releases come from different corners of the industry:
Xiaomi open-sourced Xiaomi-Robotics-0 in February. The 4.7-billion-parameter model pairs a vision-language model with a diffusion transformer that generates continuous robot movements. Xiaomi says it runs in real time on consumer-grade graphics chips.
Shenzhen-based X Square Robot released Wall-OSS-0.5 in May, trained on its own manipulation data, open-source data from many robot types and a 90-million-sample multimodal dataset. It published the model weights, training code and recipes.
Nvidia unveiled Cosmos 3, an open world model, in June to help robots and autonomous vehicles predict how the physical world behaves. Developers can build task-specific robot models on top of it.
Models are only one part of the stack. LeRobot’s flagship arm, the 3D-printable SO-101, starts at about €100, while the framework also supports humanoids such as Unitree’s G1. Hugging Face bought French humanoid maker Pollen Robotics in April 2025.
Users teach robots by teleoperation, using virtual reality devices, a second robot arm or footage filmed from their own viewpoint. The demonstrations are saved in a shared dataset format that records video alongside joint positions and other inputs, such as touch, so companies can exchange data and train on it.
Kooijmans said LeRobot closes the loop entirely, covering each step:
Building robots and recording demonstration data
Assembling the dataset and training on it
Evaluating and then running the model
Each deployed model can then generate more data, improving both the software and the robot.
Toward a million robots
Open tools are also making hard tasks achievable. In an experiment last year, Hugging Face trained a robot to fold a T-shirt thrown at it in a random state.
“This has long been a benchmark for robotics because it’s so hard to do. With all the open-source tools, anyone can collect demonstrations, train these models and do tasks like this,” Kooijmans said. “The starting point for someone else is much further along than where you have to start without all of this.”
Xiaomi has likewise tested Xiaomi-Robotics-0 on towel folding and block disassembly with a dual-arm robot.
The community has grown quickly. Shared robotics datasets have reached almost 80,000 in the past year, and growth is accelerating. He said LeRobot’s users are much more creative than Hugging Face’s own team, building different robot bodies for different tasks.
Developers keep contributing new code, ideas and models on GitHub, and companies now use the library alongside researchers and individuals.
“The industry has all these specific tasks, all these potential workflows. The research is trying to figure out what the best new algorithms are,” he said. “If we can work with more deployment data and align these two very much together, we have a better shot at solving this soon.”
He also pointed to “world models,” which let robot models learn from non-robotic data.
Nvidia’s Cosmos 3 is one such model. Developers can use it to simulate how a robot’s actions would play out in a physical environment. It also generates training data, such as joint angles and gripper positions, including for rare or dangerous scenarios like collisions. Nvidia offers it in “super” and “nano” sizes.
Kooijmans said people can prompt such models through human demonstrations rather than just text. That could make robots learn new tasks more efficiently and help scale training data with non-robotic sources.
He envisions robot models becoming as easy to download and build on as today’s open AI models. A user would simply download a robot policy, collect some data for their own task and add that capability to their own processes.
“If a million people can use and build upon this technology, then this is the fastest path to get to a million robots and more,” he said.
He summed up the shift in three points. Tasks are moving from code to data; open models and tools are pushing the starting point further along; and as the technology progresses and the ecosystem continues to grow, the time to start building is now.