Concept

Jagged Intelligence

conceptcapabilityreliabilitymulti-source
Teach me this →

Jagged Intelligence

Jagged intelligence (also described as the ‘Swiss-cheese’ model of LLM capability) is Andrej Karpathy‘s term for the uneven capability profile of large language models: brilliant across most domains, with arbitrary, often surprising gaps.

Examples

  • 9.11 vs 9.9. Early models confidently said 9.11 > 9.9. Analysis: internal activations associated with Bible verse numbering (where 9.11 does come after 9.9) were apparently co-activated, leading to the wrong comparison.
  • ‘How many R’s in strawberry?’ Models insisted on 2 for a long time. Root cause: Tokenisation — the model sees str, awberry, not individual letters, and cannot count character-by-character without additional reasoning.
  • Car-wash example (Karpathy, 2025): ‘I want to go to a car wash 50 metres away. Should I drive or walk?’ State-of-the-art models say to walk — correct on the surface, but missing that you need a clean car at the other end. A model that can refactor 100,000-line codebases gets tripped up by this.

Why jaggedness exists

Several structural causes:

1. Verifiability

The RL training loop requires a reward signal, which requires a verifier. Labs built RL environments around maths and code because those domains are economically valuable and technically clean. Domains without easy verification — aesthetic judgment, common-sense reasoning about goals — received less RL investment and show more gaps.

2. Data and attention

Even within verifiable domains, capability reflects what the labs put in the training mix. Chess improved dramatically between GPT-3.5 and GPT-4 because someone at OpenAI added a large chess corpus to pretraining. The decision to focus creates a capability spike. Absence of focus leaves a gap.

3. Eval-shaped RL (Ilya Sutskever)

In Ilya Sutskever on the Age of Research, Continual Learning, and Alignment, Sutskever offers a complementary account of jaggedness. His vivid example: a vibe-coding model fixes bug A by reintroducing bug B, then fixes B by bringing back A, oscillating forever — alongside superhuman eval scores. Two causes. The whimsical one: RL training makes models ‘too single-minded and narrowly focused’. The structural one: where pre-training never forced a choice of data (‘the answer was everything’), RL forces teams to hand-build environments, and they ‘inadvertently’ take inspiration from the benchmarks — so eval scores climb while real-world performance lags. As Dwarkesh Patel put it and Sutskever endorsed, ‘the real reward hacking is the human researchers who are too focused on the evals.’ His student analogy: the model is the 10,000-hour competitive programmer with every technique at its fingertips, who generalises worse than the 100-hour student who has ‘it’. The deeper driver is the Generalization Gap — models generalise dramatically worse than people. See also Evals.

4. Off-manifold novelty

In Andrej Karpathy on AGI Timelines, RL's Limits, and the Future of Education, Karpathy locates a deeper edge of the jaggedness: models are strong on patterns common on the internet and weak on work that has never been written before. Building nanochat, agents kept forcing the conventional pattern (the PyTorch DDP container he had deliberately replaced), bloated the code with defensive boilerplate, and used deprecated APIs — ‘it’s slop.’ He attributes this partly to memory: models memorise too well and are ‘distracted’ by the training manifold, where humans’ weaker recall forces them to generalise. His proposed remedy is the cognitive core — stripping memorised knowledge to keep the algorithms of thought. He likens current models to ‘a kindergarten or elementary-school student’: savants with perfect recall who can pass PhD quizzes yet remain cognitively immature.

Practical consequences

  • Do not treat LLMs as reliable on any task without testing.
  • Find where the capability holes are for your specific application — there is no manual; you have to explore empirically.
  • If your application lives outside the circuits that were activated by RL training, you will need fine-tuning or a domain-specific setup; the base model out-of-the-box won’t get you there.
  • Always verify outputs — especially for arithmetic, character-level operations, and multi-step reasoning.

See also