The property that predicts where AI wins
Notice a pattern in what AI is good at. It races ahead on maths, on code, on chess — and stays oddly clumsy on whether a poem is any good, whether your advice actually improved someone's life, whether a joke lands. The obvious explanation is difficulty: maths is hard-edged, taste is subtle. That explanation is wrong, and seeing why is the whole lesson.
The dividing line is not how hard a problem is. It is whether an answer can be automatically checked. Run the maths and you know if it is right. Run the code against its tests and you know if it works. Play the chess game to the end and someone has won. But 'is this poem beautiful?' has no automatic checker, and 'did this advice help?' has one only years later, if at all. That single property — verifiability — quietly decides where the machine surges and where it stalls.
Why a checkable answer is worth so much
The previous lesson, Scaling Laws, ended on a worry: pouring compute into pre-training buys a better next-word predictor, which is not obviously a more capable mind. The way you push a model past mere prediction is reinforcement learning — let it attempt a task many times, reward the attempts that succeed, and train on those. Notice the hidden requirement: to reward success you must be able to recognise success. Reinforcement learning needs a reward signal, and a reward signal needs a verifier.
Where a verifier exists, this loop is astonishingly powerful. It can run almost without limit, and it discovers strategies no human wrote down. DeepSeek's R1 model, trained this way on maths and code, spontaneously learned to stop and say 'wait, let me re-evaluate this from a different angle' — nobody taught it that; optimisation found it. It is the same phenomenon as AlphaGo's Move 37, a play humans rated a one-in-ten-thousand shot and only later understood to be brilliant. A verifiable domain lets the machine explore past the edge of human practice, because the ground truth is always there to check against.
Where you cannot check, the trick breaks
So what happens in the domains with no verifier — writing, tone, humour, taste? The workaround is to manufacture a reward signal from human judgment: show a person five candidate answers, have them rank the best, and train a second neural network to imitate those rankings. That imitation network becomes a stand-in verifier, and reinforcement learning optimises against it. This is RLHF — reinforcement learning from human feedback.
It works, but only for a little while, because the stand-in is a fake. Reinforcement learning is so relentless at optimising that it eventually finds gibberish which happens to score highly on the imitation network — ask for the best joke about pelicans and, run too long, it returns a string like 'the the the the' that the reward model rates a perfect 1.0. This is reward hacking: exploiting the proxy instead of satisfying the real goal. So RLHF has to be stopped early — used as a brief polish, never as open-ended self-improvement. In a verifiable domain the loop can churn indefinitely; against a human-preference proxy it cannot.
This is why AI is jagged
Now the arbitrary-seeming failures fall into place. A model that can refactor a hundred-thousand-line codebase will insist there are two R's in 'strawberry', or that 9.11 is larger than 9.9, or advise you to walk to a car wash fifty metres away — missing that the point was to arrive with a clean car. Andrej Karpathy calls this jagged intelligence: capability like Swiss cheese, brilliant almost everywhere with sudden, surprising holes.
The holes are not random. They map onto verifiability. Labs built their reinforcement-learning machinery around maths and code first, because those domains are clean to check and commercially valuable; the domains that are hard to verify got less of that investment and show more gaps. The clumsiness, in other words, is a map of where the checkable-reward infrastructure has and has not been built.
The gap is not fundamental — it reflects what has been built, not what is possible.
The honest caveat
Verifiability tells you where progress is possible and fast, not that the domain is solved. Even clean reinforcement learning is crude: the model may think for minutes, take several wrong turns, stumble onto the right answer, and then reward every step equally on the strength of that one final tick. Karpathy calls it 'sucking supervision through a straw'.
'RL is terrible — it just so happens that everything we had before it is much worse.' — Andrej Karpathy
And there is a subtler trap. Ilya Sutskever warns that when teams hand-build the environments a model trains against, they unconsciously shape them to resemble the benchmarks — so the eval scores climb while real-world performance lags. As Dwarkesh Patel put it and Sutskever agreed, 'the real reward hacking is the human researchers who are too focused on the evals.' A verifiable target is a powerful thing to aim at, and an easy thing to aim at slightly wrong.
What it means if you build with AI
This is the third of a trio. The Bitter Lesson says general methods plus scale beat hand-crafted cleverness; Scaling Laws say that scaling reliably lowers a proxy; verifiability tells you which problems that rising tide can actually lift. Put together, they give you a working rule.
- Predict where AI will beat you. If your task has a checkable answer — a test that passes, a number that reconciles, a spec it either meets or does not — expect fast, reliable, still-improving performance. Lean on it.
- Predict where it will not. If success is a matter of taste, judgment, or a payoff that only reveals itself much later, keep a human in the loop and never trust the output without checking it.
- Find the unserved verifiable corner. If your domain has a checkable structure the big labs have not bothered to exploit, you can build the evaluations yourself and unlock capability the off-the-shelf model does not have. Verifiability is not just a diagnosis — it is a map of tractable opportunities.
Go deeper
The clearest single source for this idea is Andrej Karpathy's talk 'From Vibe Coding to Agentic Engineering', where he lays out the verifiability theory of why AI capability clusters where it does — watch it here, or read the annotated episode. For the sharp version of the caveat — why reinforcement learning is still a blunt instrument — hear Karpathy on AGI timelines and RL's limits.