Ocado Group, a Hatfield-based grocery technology firm, is finding that a robot error too rare to notice in a lab becomes a weekly problem at scale.
Its robotic arms now pick about 10 million grocery items a week. A drop rate of one in 100,000 sounds like a rounding error, but it means around 100 falling items a week, each able to halt a bot, summon an engineer and dent uptime.
“At scale, a one-in-a-million edge case is not an edge case. You have to design your system and your processes assuming that the worst will happen,” said Matt Whelan, vice-president of engineering for automation at Ocado.
“Making the robot do the thing is only a small part of the problem. The big part is everything around it and the costs it can quietly push into different parts of your operation if you’re not paying attention.”
Whelan said physical artificial intelligence (AI) suddenly seems to be moving very quickly, and many people, himself included, feel a “ChatGPT moment” for robots is close. He expects many organizations to buy or build robots that look brilliant in a demo, only to discover how hard they are to run at scale.
Internally, Ocado uses the term “serviceable operational range” for what an arm can reliably pick in a real environment without causing problems. Speed and total addressable range, the obvious questions a buyer asks, miss what matters.
“You can get your robot metrics looking great, but the cost has moved somewhere else in the system. I’ll leave you to imagine the impact of spilling coconut oil on the grid. We know from experience,” he said.
The grid stores crates eight meters high and is served by washing machine-sized bots that can reach any stack.
Since deploying its first arm at the end of 2023, Ocado has gathered a growing dataset of edge cases and failures, along with human demonstrations of exemplary and recovery behavior. It uses the data to train and refine the arm’s controller.
Ocado was founded in 2000 as an online grocer whose warehouse staff pushed trolleys for miles down aisles. It now runs its automated system as a platform for retail partners.
The £5 double pick
Whelan spoke at the Physical AI and Robotics Summit, organized by Economist Enterprise in London on October 13. His case-study session asked how companies can tell whether automation adds value or merely shifts costs from one part of a business to another.
A double pick, where a suction cup lifts two items instead of one, still counts as a robot success.
“Its KPIs (key performance indicators) are looking fantastic. If you’re in a grocery business with very low margins, you save a few pence per pick on everything you do successfully, then you give away a £5 item. You’ve undone all the value of hundreds of picks with one mistake,” Whelan said.
Because Ocado runs a platform business, it must weigh the arm’s impact on its partners’ economics as well as its own. Its costs include robotic damage, reliability hits when items land on the grid, arm repairs and remote support.
“Your robot labor savings sits on one side of the ledger, and everything else sits on the other. I improved the range that my robot could pick. My double-pick rate went up, and my overall P&L (profit and loss) got worse,” he said.
Whelan said companies should map everything their automation touches upstream and downstream, measure success across the whole operation and, whenever one metric improves, find out what quietly got worse.
Ocado’s task is relatively constrained: the same box size and the same goal every time, pick an item and place it carefully in a customer’s bag. But it unfolds in a complex industrial setting, with messy real-world challenges, in a low-margin industry.
“We are not trying to solve a fully generalized robotics problem. We’re trying to do one hard thing extremely well in a punishing industry,” he said.
Keeping the arms running is a job in itself. Ocado supports every arm worldwide from a single center, a 24/7 operation spanning time zones, peaks, troughs, and shift patterns, and its staff must be able to take remote control when a robot misbehaves.
“There is no such thing as a 100% success rate in hardware,” he said.
Range management, deciding which items a robot should touch, is another moving target as suppliers change packaging without warning.
Releases in harm’s way
Whelan’s answer is to learn in production. Ocado rolls changes out to a subset of arms, A/B tests them, deploys them gradually across sites and rolls them back if needed, with safeguards to match.
“You cannot find the one-in-a-million edge cases in a lab. You have to get into the real, chaotic world as early as possible, and that means your latest release is always in harm’s way. When you have hundreds of arms deployed, you can learn fast without putting the operation at risk,” he said.
He said this is how companies can move fast and break nothing that matters, even though breaking nothing at all is impossible in the physical world. He also urged a relentless focus on total cost of ownership until the system crosses its break-even line.
Beyond that point, better capability and growing scale can compound. Whelan said Ocado’s arms are approaching half a billion picks and should pass one billion around this time next year. For partners, the system cuts the labor needed to pick and pack a grocery order by about half.
The robotic picking system now runs for retailers in eight of Ocado’s markets, including the UK, the US, Japan and South Korea, and the company aims to push that labor saving higher.
Whelan said AI is moving robotics from traditional systems toward more generalized, more capable automation, and that getting a few aluminum tubes with actuators at the corners to perform a diverse task well is extraordinary. Such systems also need warehouse integration, remote-control interfaces and cloud infrastructure.
“It’s not just the control model either. It’s the perception layers on top that look for errors and intervene in the robot control without needing a human in the loop,” he said.
Classical robotics, such as the cells that weld Ocado’s metal totes, makes high-value goods and enjoys forgiving economics.
“You could build a super impressive system and never make a penny from it if you start with wrong assumptions. The robot that used to fit a car door is now trying to do something trivial at home. We are looking for much lower-value tasks at much higher volume,” he said.
He said grocery offers a preview of the problems the wider industry will face as physical AI spreads into everyday tasks, where every double pick and minute of remote intervention will be felt.