Over-Attribution of Intelligence

Over-attribution of intelligence is the human tendency to ascribe understanding, reasoning, and general intelligence to a machine on the strength of fluent output, when the underlying system has none of those things. The term is Gary Marcus‘s; the phenomenon was first documented by Joseph Weizenbaum in the 1960s.

The idea draws a line between two questions that are easy to conflate: does the system produce human-like output? and does the system possess the capacities that produce that output in a human? A system can pass the first test while failing the second completely. Weizenbaum’s ELIZA, a 1960s program that mimicked a psychotherapist by matching keywords (‘tell me more about that relationship’), led users to perceive empathy and understanding that were not present. Marcus’s claim is that the same vulnerability, scaled up, now sits under the entire AI market: large language models produce fluent text by predicting the next token, and observers read fluency as intelligence.

The cognitive-science argument

Marcus grounds the concept in why humans are bad judges of machine intelligence. Evolution equipped us to detect fast-moving threats — snakes, predators — but ‘we have nothing built into our brain to really help us think about the nature of intelligence’. Because LLMs are explicitly built to mimic human output, and humans are not built to distinguish a mimic that works on different principles from the real thing, the misjudgement is close to automatic. The corrective is technical: a proper test of intelligence requires the cognitive-science background to know what to measure — instruction-following, generalisation outside the training distribution, reliable reasoning — rather than reading surface fluency as proof.

Comparative cognition reaches the same conclusion from a different discipline. The cognitive scientist Erica Cartmill, who studies minds that cannot be interrogated in words — great apes, human infants — argues that a verbal answer is not evidence of the process behind it: ‘you can ask an LLM and it will give you an answer, but the question is, does that answer map onto why they did that?’ Her corrective is methodological rather than merely a matter of picking the right benchmark. Because verbal self-report is unreliable even in human adults — ‘the reason we think we did something might not actually be the reason we did it’ — she proposes importing the toolkit built for non-verbal minds: probe understanding by manipulating the modality and the framing of a task, not by taking the system at its word. On her account the field will not engineer its way past this; assessing what a model understands needs the same triangulation of behaviour, not-fluency, that animal-cognition and developmental psychology already use.

The concept is distinct from its neighbours. Jagged Intelligence describes a property of the systems — genuine capability that is unevenly distributed, brilliant in some domains and absent in others. Over-attribution describes an error in the observer — wrongly inferring general intelligence from narrow or merely apparent competence. The two interact: jagged capability makes over-attribution easy, because the impressive cases are salient and the failures are out of view until probed.

Why it matters for markets

Marcus’s reason for naming the error is that it is load-bearing for the economy, not merely a philosophical point. If investors over-attribute intelligence, capital flows to a paradigm on the basis of a capability that is not there — ‘people betting trillions of dollars that these machines are intelligent in ways that they aren’t actually’ — because the people placing the bets lack the test to evaluate the claim. The concept thus links the cognitive-science critique to the financial thesis: misjudged capability underwrites misjudged valuations.

Where mainstream views differ

The mainstream lab view inverts the charge: rather than observers over-attributing intelligence, current systems under-display an intelligence that scale and better harnesses will soon reveal — today’s models being ‘the worst you will ever use’. On that reading, fluent output is an early signal of real general capability, not a mirage, and the apparent failures are temporary artefacts. The disagreement turns on whether the gap between output and understanding is closing with scale (the maximalist position catalogued in The Road to AGI) or is structural to next-token prediction (Marcus’s position, shared in part by Yann LeCun).

In the wiki

• Gary Marcus on AI's Overstated Intelligence, LLM Economics, and the Limits of Scaling — where Marcus sets out the concept
• Gary Marcus — who named it
• Erica Cartmill on Animal Minds, Play and Teasing, and Comparative Cognition — a comparative-cognition scientist reaches the same conclusion: verbal output is not evidence of the process behind it, and probing an LLM needs the methods built for non-verbal minds
• Jagged Intelligence — a property of the systems, not the observer; easily confused
• Hallucination — a related failure that over-attribution leads observers to discount
• The Road to AGI — the debate over whether the output–understanding gap closes with scale