The finding
For a decade, making an AI model better was a craft: try a new idea, hope it works. Then researchers noticed something that turned the craft into arithmetic. The performance of a large language model on its core task — predicting the next word — could be read off in advance from just two numbers: how big the model is (its parameter count) and how much text it was trained on. Make either bigger and performance improves — smoothly, predictably, and with no sign yet of levelling off.
The startling part is what the finding leaves out. You do not need a new insight. You do not need a cleverer method. Scale alone buys the improvement. When OpenAI went from GPT-3.5 to GPT-4, nearly every benchmark rose — not because of one breakthrough, but because the model was bigger and had seen more.
Why it changed the whole industry
A smooth, predictable curve is a licence to spend. It converted AI from unpredictable research into an engineering project you could budget: put in more compute, get out a better model. That single fact reorganised an industry. It is why NVIDIA became one of the most valuable companies on earth, why firms pour billions into data centres, and why the race is for more chips and more data rather than for one elegant idea.
'Elon Musk getting 100,000 GPUs in a single data centre? They're all doing this — predicting the next token, faster.' — Andrej Karpathy
It even shaped government policy — the United States bet that if capability tracks compute, denying China the most advanced chips would deny it the frontier. And it shaped how products are built. Sam Altman's standard line, that today's models are 'the worst AI models you will ever use', is the scaling bet turned into advice: whatever the model cannot do today, it will do soon, so build for the version that is coming.
It's not one dial, but three
The early story was a single dial: make the pre-trained model bigger. The sharper picture, set out by Nathan Lambert, is three independent dials — all still turning.
- Pre-training. The original dial: a bigger model on more data yields lower loss. It buys permanent capability at a fixed, one-off cost. Its returns are slowing, but it is not dead.
- Reinforcement learning on verifiable rewards. Reward the model for getting a maths problem or a piece of code demonstrably right. Cheaper per unit of capability, and the engine behind the reasoning-model breakthroughs of 2025.
- Inference-time compute. Let the model think for longer at the moment you ask it a question. The cost accrues per query rather than once up front, and there is no obvious ceiling.
The lab that prices these trade-offs correctly wins — not necessarily the one with the most raw compute.
Where the bet is contested
Here is the subtlety that decides everything. The curve is real, but look closely at what it measures: loss on next-word prediction. A model can get steadily, reliably better at guessing the next word while the things you actually care about — sound reasoning, not inventing facts, coping outside its training data — barely move.
That gap is the sceptics' whole case. Gary Marcus argues that scaling the metric does not buy the capabilities that matter, and that hallucination is structural rather than a flaw you can scale away. Ilya Sutskever — who co-authored the canonical results the scaling thesis rests on — puts the puzzle plainly: 'these models somehow just generalise dramatically worse than people.' And there is an economic sting: if every lab climbs the same curve with the same recipe, the result is commoditisation — no moat, no profit.
So the honest reading is narrow. The smooth curve guarantees a better next-word predictor. Whether that is the same thing as a more capable mind is exactly what the field is fighting about.
What it means if you build with AI
This is the companion to a related idea, the Bitter Lesson — that general methods fed with scale beat hand-crafted cleverness. Put the two together and a working rule falls out.
If you trust the curve, build for the more capable model six months out: keep your scaffolding light, and do not over-engineer around limits that scale is about to erase. If you lean sceptic, invert the discipline: do not stake the whole enterprise on the extrapolation continuing, and test whether the specific capability you depend on actually improves with scale — or whether only the benchmark does. Both camps agree on the one thing that matters here: the curve measures a proxy, and a proxy can quietly come apart from the thing you were really after.
Go deeper
The clearest single explanation of how scale produces capability — pre-training, and the newer reinforcement-learning and reasoning dials layered on top of it — is Andrej Karpathy's 'Deep Dive into LLMs like ChatGPT' — watch it here, or read the annotated episode. For the sharpest version of the case against, hear out Gary Marcus on why the curve may be measuring the wrong thing.