Ilya Sutskever on the Age of Research, Continual Learning, and Alignment

Source:
Ilya Sutskever on the Age of Research, Continual Learning, and Alignment
transcriptdwarkesh-podcastagiscalingssialignmentcontinual-learninggeneralisation

Ilya Sutskever on the Age of Research, Continual Learning, and Alignment

Ilya Sutskever in conversation with Dwarkesh Patel. Published November 2025. Ilya argues that the age of scaling is over and we have re-entered an age of research. The fundamental unsolved problem is generalisation: models perform superhuman on benchmarks but generalise dramatically worse than humans. The path to superintelligence runs through continual learning — models that learn from deployment the way humans learn from experience — and the right alignment target is an AI that cares for sentient life.

Key ideas

  • From the age of scaling to the age of research. Pre-training scaling (2020–2025) was predictable but is plateauing. The next breakthroughs require genuine research ideas, not more compute applied to the same recipe. “We are squarely an age of research company.”
  • Models generalise dramatically worse than humans. The fundamental problem: a model can be superhuman at competitive programming yet fail trivially at tasks that reflect general intelligence. Ilya attributes this to the narrow environments used for RL training — the competitive-programmer student analogy vs. the 100-hour generalist student.
  • Value functions and emotions. Human emotions function as an evolved value function: a simple but extremely robust signal that enables self-directed learning without external reward. The student who learned to drive in 10 hours uses their internal sense of how badly they are doing, not an externally provided score. Models currently lack this.
  • Continual learning as superintelligence. Not a fully pre-trained omniscient system but a model that can learn from deployment like a human joining a job. The “superintelligent 15-year-old” — eager, capable of learning anything, but not omniscient on day one.
  • AGI and pre-training as limiting terms. “AGI” arose as a reaction to “narrow AI.” “Pre-training” trained everyone to expect general capability from a single training phase. Both concepts may be misleading: real superintelligence will probably involve continual learning post-deployment, not a finished artefact.
  • Alignment: care for sentient life. More tractable than specifying human values because the AI itself will be sentient; empathy-like properties may emerge. The first N powerful systems need to care about sentient life — if that is achieved, a good path follows.
  • 5–20 year timeline to systems that learn as well as humans.

The Age of Research

Ilya’s periodisation: 2012–2020 was the age of research (ideas mattered, compute was the bottleneck). 2020–2025 was the age of scaling (compute allocation dominated, the recipe was known). 2025+ is back to research — the recipe has been applied, the models are large, but the remaining gains require new ideas. The word “scaling” has shaped all thinking because it tells companies what to do: add more compute. In the age of research, the directive is less clear, and the companies with genuine insight will pull ahead.

Why Models Don’t Generalise

Ilya’s account of the generalisation gap:

  • RL training environments are deliberately designed to produce good benchmark performance — this is a form of reward hacking at the organisational level, not just the model level.
  • The competitive programmer analogy: models trained extensively on narrow environments become very good at those environments and worse at others. The 10,000-hour student, not the 100-hour intuitive one.
  • The solution is not more diverse RL environments (more competitive programmers) but a fundamentally different approach to learning that achieves reliable generalisation from limited experience.
  • Value functions — internal estimates of how well or badly you are doing — are the missing ingredient. The student learning to drive has an internal signal; current RL models do not, or have a crude approximation.

See also