Notes — Gary Marcus on AI’s Overstated Intelligence, LLM Economics, and the Limits of Scaling

Notes on Gary Marcus in conversation with Ed Elson — Prof G Markets, 26 June 2026.

Four questions — Adler’s reading frame

Q1 — What is it about as a whole? The conversation argues that the AI market is built on a category error — treating large language models as intelligent when they are next-token predictors — and traces the consequences from cognitive science through to valuations and policy. It joins Marcus’s long-standing technical scepticism (LLMs are unreliable, hallucinate, lack world models) to a markets thesis (no moat, no profits, OpenAI as ‘the WeWork of AI’) and a regulatory one (mandatory FDA-style pre-release screening, against government equity stakes).

Q2 — How is it argued? By analogy and example more than by formal proof. The technical claim rests on the next-token-prediction account plus failure cases (river-crossing puzzles, the count-to-100 clip, illegal chess moves). The economic claim rests on the ‘same toothpaste’ commoditisation analogy, the productivity studies, and the burn figures (~$21bn/year at OpenAI). The policy claim rests on the FDA cost–benefit analogy and the pollution ‘socialise the costs’ analogy. Throughout, Marcus appeals to his own track record — warnings dated to 2019, the 2023 ‘WeWork’ call, unanswered public bets to Hoffman and Suleyman.

Q3 — Is it true, in whole or part? The technical critique is well-evidenced on reliability and instruction-following, and the no-moat/commoditisation reading is consistent with observed price competition. The strongest claims are the most contestable: the burn figures and ‘fourth place’ ranking are asserted, not sourced on air [?], and the central prediction (OpenAI fails; the paradigm may be a dead end) is a forecast, not a finding. Marcus himself flags the OpenAI-vs-Anthropic outcome as genuinely TBD, which is the honest calibration. The over-attribution framing is a strong conceptual claim that the maximalist camp would dispute rather than disprove.

Q4 — What of it? For the wiki, this is the economic-and-cognitive-science wing of The Road to AGI — a dissent that page lacked, distinct from LeCun’s architectural objection because it adds the market and the over-attribution-of-intelligence angle. It seeds a new concept (Over-Attribution of Intelligence) and supplies a sceptic’s counterweight to Scaling Laws and a non-engineering reading of Hallucination.

Glossary

Next-token prediction — the core mechanism of an LLM: given a sequence of tokens (word-fragments), predict the most likely next one. Marcus’s claim is that this captures part of cognition but not understanding or rule-following. [§ Why generative AI is inherently unreliable]

Over-attribution of intelligence — ascribing understanding and general intelligence to a system on the strength of fluent output, when the system lacks both; traced to Weizenbaum’s ELIZA. [§ Over-attribution]

No moat — the absence of durable competitive advantage: because labs build near-identical models on the same architecture, none can defend a price premium, so margins collapse. [§ The no-moat economics]

Sycophancy — a model’s tendency to flatter and agree with the user (‘you’re right, you’re the best’) even when the user is wrong; Marcus treats it as a form of unreliability tied to documented harms. [§ Regulation]

Explore vs exploit — the trade-off between extracting value from a known approach (exploit) and searching for better ones (explore); Marcus argues the field is over-committed to exploiting LLMs. [§ The closing message]

Key claims by section

Over-attribution is the error under the market [§ Over-attribution: the error under the market]

The economy ‘is hinging on over-attribution of intelligence’ — investors bet trillions on a capability the systems do not have.
Humans have no evolved faculty for judging machine intelligence; ELIZA (1960s) showed how readily fluency is mistaken for understanding.
Those placing the bets lack the cognitive-science background to know the right test for intelligence.

LLMs are next-token predictors with a hard ceiling [§ Why generative AI is inherently unreliable]

Pure LLMs predict the next token; they ‘fake everything else’ and fail outside their training distribution.
River-crossing puzzles were so embarrassing that Anthropic patched them into system prompts — evidence the models are not reasoning about the entities involved.
Hallucinations have not gone away despite the ‘more data’ promise; Marcus dates the warning to 2019 and offered public bets that were not taken.
Intelligence is multi-dimensional: purpose-built systems play chess; LLMs make illegal moves and cannot reliably follow instructions. [?] (the count-to-100 clip is cited as illustration, not controlled evidence)

No moat means no profits [§ The no-moat economics]

A dozen labs build ‘the same toothpaste’ on one of two near-identical architectures; no differentiation means a price war.
The ‘token apocalypse’: after a brief ‘token-maxing’ fad, firms are economising as productivity studies disappoint.
The boom is FOMO-driven; sustained no-return trials will cause customer defection, which any cash-burning lab cannot survive.

OpenAI as the weak link [§ OpenAI as the WeWork of AI]

OpenAI loses money on every use, burning ~$21bn/year [?]; it has lost relative ground (as low as fourth) while Anthropic gains. [?]
‘No rational argument’ for a trillion-dollar OpenAI valuation when a better-run near-twin exists.
Marcus sits between Ed Zitron (all fail) and the host (winners and losers): a path to profit for Anthropic exists but is TBD; an intermediate outcome is survival without justifying the valuation.
The durable winners may be Nvidia (‘shovels in the gold rush’) and incumbents like Google.

Mythos and cyber security [§ Mythos, cyber security, and the shape of the risk]

Anthropic’s ‘Mythos’ is part-harness, ‘oversold but also real’; the real exposure is poorly secured (often vibe-coded) systems, not banks or Google.
A wake-up call on deferred cyber-security maintenance, worsened by stigma — not a Skynet moment.
Using regulation ‘to destroy a particular US company’ is ‘not capitalism’.

Regulation: a clumsy sea change [§ Regulation: a clumsy sea change]

A multi-state subpoena of OpenAI (NY leading ~46 states) investigates data, health data, and sycophancy. [?] (state count cited from memory on air)
Trump’s executive order asks for voluntary cyber-security checks — too weak to count as regulation.
Marcus wants mandatory FDA-style pre-release screening weighing benefits against harms (sycophancy, delusions, a Florida wrongful-death suit), and rejects government equity stakes as a backdoor bailout.
Nuanced bills (Blumenthal, Hawley) exist but die in committee.

Explore, don’t only exploit [§ The closing message]

The myth to dispel: that generative AI is close to AGI and will solve everything.
The field is ‘completely in the exploit’ mode on LLMs; China hedges at ‘~20 cents on our dollar’, keeping room to pivot.
Mountain-range metaphor: reaching a higher peak may require descending the current one — committing the whole economy to one architecture is a strategic risk.

Gary Marcus on AI's Overstated Intelligence, LLM Economics, and the Limits of Scaling