Sergey Levine on Robot Foundation Models, Generalisation, and Embodied AI

Sergey Levine, co-founder of Physical Intelligence and a UC Berkeley professor, talks with Patrick O’Shaughnessy about building robotic foundation models — a general-purpose ‘brain’ that can control any physical robot to do any task. He explains why robotics has lagged behind language and image models, how the field is borrowing and adapting the recipe that made those models general-purpose, and what a working embodied foundation model would unlock for the people building robots.

Key ideas

1. Robotics has a ‘scarecrow problem’. Physical robots keep getting more capable as hardware, but they lack a general intelligence to run them. Physical Intelligence’s goal is a robotic foundation model that can ‘control basically any embodied system to do any task’ — a shared brain that any body can plug into, rather than a bespoke controller built per robot per task.
2. The language-model analogy, and why it holds. Language models overtook narrow, domain-specific systems (hand-built machine translation, sentiment analysis) not just by pooling more data but by mining weakly labelled data at web scale — text nobody curated for a specific task. That builds a broad foundation of world understanding, on top of which narrower applications become easy to bolt on. Levine argues the same logic applies to robots: a model that understands physical interaction in general needs far less data to pick up any one new skill than a model trained only on that skill (a ‘dishwashing specialist’ or ‘laundry-folding specialist’).
3. Effective generalisation is not what makes an exciting demo. Levine calls this tension a recurring problem across his career: the way to make a compelling robot demo is to pick one cool task and control everything else in the environment until it is pristine. Generalisation instead means doing something mundane — Physical Intelligence’s kitchen-cleaning demo was ‘just picking up plates’ — in a setting the robot had never trained on, which reads as far less impressive out of context than it is.
4. Success means unlocking other people’s imagination, not just crossing a capability line. Levine compares the payoff to the personal computer: today, a new robotics idea requires building an entire intelligence stack from scratch. A general-purpose foundation model that people can prompt, fine-tune, or lightly adapt removes that barrier, the way PCs (and later the internet) let far more people build far more things. He expects robots to become a toolkit rather than ‘one thing’ — arms, quadcopter swarms, humanoids, and shapes nobody has built yet, all running on the same underlying model.
5. Common sense comes from chain-of-thought, skill comes from reinforcement learning. Physical Intelligence’s models start as vision-language-action models — an LLM adapted first with web image data, then with diverse robot data. Chain-of-thought reasoning (‘pick up the plate’) lets the model draw on that web-scale prior knowledge to handle long-tail situations it has never directly practised. Reinforcement learning then improves speed, robustness, and throughput on top of that base, illustrated by a robot that learned to make espresso through repeated practice.
6. Robot data cannot simply mirror the internet, so the plan is to bootstrap. There is no internet-scale dataset of robot experience, and Levine doubts anyone actually needs to quantify how much would be enough. The plan instead is to get a robot useful enough that it can go out into the world and gather more data through its own deployment — the same flywheel Tesla gets from cars already on the road. He also notes the bottleneck has shifted: models now improve more from added semantic, high-level supervision (labelling what a robot already did) than from more low-level teleoperation data.

Context

Patrick O’Shaughnessy discloses at the outset that he is an investor in Physical Intelligence. The conversation ranges well beyond the points above — Levine also discusses Moravec’s paradox in robotics, the real (and much-debated) split between simulation-heavy and real-data-heavy training pipelines, why elderly and childcare tasks may prove harder than they look, and how coding tools offer the closest existing template for how AI will change labour in physical jobs.

See also

• Patrick O'Shaughnessy — host
• Caitlin Kalinowski on Building Hardware, the Robotics Frontier, and Supply-Chain Reindustrialisation — a hardware-side view of the same robotics frontier
• Andrej Karpathy on AGI Timelines, RL's Limits, and the Future of Education — Karpathy’s parallel scepticism about reinforcement learning’s limits and where genuine generalisation comes from
• Dylan Patel on the Token Economy, AI Supply-Demand, and the Permanent Underclass — the compute and supply-side backdrop against which foundation-model bets like this one are made
• Jensen Huang on NVIDIA, AI, and the Future of Computing — Huang’s physical-AI thesis that humanoid robots will use existing tools rather than have the world rebuilt around them
• Large Language Models — the architecture and scaling story Levine’s analogy draws on
• Scaling Laws — the broader-data-beats-narrower-data logic Levine applies to robotics