Anil Seth on Consciousness, Machine Sentience, and Why Intelligence Isn’t Awareness

Neuroscientist Anil Seth returns to argue that the case for conscious AI rests on a confusion — treating intelligence (doing) and consciousness (feeling) as one thing — and that consciousness is far more likely to be a property of living organisms than of any computation, however fluent.

Key ideas

1. Intelligence is about doing; consciousness is about feeling and being. The two travel together in us, but that is a fact about living creatures, not a law that binds all minds. Certain experiences (regret) require certain cognitive competences (envisaging alternatives), yet nothing shows that being cognitively capable must be accompanied by experience. The whole debate about conscious machines, Seth argues, mistakes their evident intelligence for evidence of awareness.

2. Computational functionalism is a bet, not a starting point. The claim that AI could be conscious is the stronger claim that consciousness happens in virtue of computations — substrate-independent mappings of symbols to symbols. That has to be earned, not assumed. Turing computation cannot do everything (continuous and random processes escape it), and it is unclear it can realise all the properties of a brain.

3. You cannot separate what a brain is from what it does. Computers are useful precisely because software floats free of hardware; evolution built brains under the opposite pressure, because insulating one scale of dynamics from another is energetically expensive. Seth calls this entanglement generative entrenchment — and it is why the standard neuron-replacement thought experiment quietly falls apart: some neurons fire spikes to clear metabolic waste, so a truly equivalent silicon neuron would need a metabolism, at which point you have not built a computer but a biological being.

4. Consciousness may require being alive. Seth defends a refined biological naturalism, dissatisfied with the bald identity (‘consciousness is just what living systems have’) and reaching for positive reasons: a line running from the brain as a prediction machine (controlled hallucination) down to metabolism and autopoiesis — living systems maintaining themselves out of thermodynamic equilibrium — bridged, tentatively, by Karl Friston’s free-energy principle.

5. Our biases manufacture false positives. Human exceptionalism, the conflation of intelligence with consciousness, and the special seduction of language make us over-attribute experience to LLMs while never suspecting AlphaFold, which runs on nearly identical machinery. The stripping-away test: if removing a quality (sight, memory, a sense of humour) from a human leaves them still conscious, that quality cannot be a sufficient sign of consciousness in a machine.

Summary

Feeling versus doing

Seth opens on the distinction his prize-winning essay, The Mythology of Conscious AI, is built to protect: intelligence is the capacity to do things — solve problems, act functionally — whereas consciousness is ‘primarily about feeling and being rather than doing’, Thomas Nagel’s sense that there is something it is like to be an organism. In living creatures the two are tightly related; some conscious states presuppose cognitive machinery. But relation is not identity, and the confident modern assumption that a sufficiently intelligent system must also feel is exactly the unexamined move the essay sets out to dismantle. Alex O’Connor presses the framing: we reason from the one case we know — ourselves, where intelligence and awareness coincide — and wrongly generalise the pairing.

Against substrate-independence

The engine of the pro-AI view is computational functionalism: consciousness as a property of computations, which by definition are independent of what implements them. Seth grants the thought experiment its usual run — replace one neuron with a functionally perfect silicon equivalent, then ten, then all 86 billion, and where would consciousness go? — then attacks its central assumption. Ned Block’s reply is that experience might simply fade as the substrate changes, so the argument begs the question. Seth’s deeper objection is that the premise is barely conceivable: a neuron that fires to clear metabolic waste cannot be matched by silicon unless the silicon is given a metabolism, and the more faithfully you replicate a neuron the more you are simply building a neuron. What looks like engineering a conscious computer is really assembling a ‘Frankensteinian biological being’ and calling it artificial.

The bridge Seth is not yet satisfied by

Pushed on how strong his verdict is, Seth is careful: ‘I’m very skeptical, but… we need a certain residual humility.’ Anyone who declares conscious AI either certain or impossible overstates the case. His positive story runs from perception as inference — the brain as a prediction machine issuing a ‘controlled hallucination’, its best guess at the causes of sensory signals — down through the free-energy principle to the thermodynamic facts of being alive: metabolism, autopoiesis, a living system continually producing itself and holding itself out of equilibrium with its environment. The hard step, which he openly flags as unfinished, is joining information-theoretic free energy (prediction error) to thermodynamic free energy (the energetics of metabolism). He offers it not as proof but as at least as plausible as the computational bet.

Why LLMs fool us and AlphaFold does not

Seth locates the credulity in us, not the machines. Three biases do the work: human exceptionalism, the conflation of intelligence with consciousness, and the pull of language. AlphaFold, architecturally close kin to a large language model, never tempts anyone to worry about its inner life; GPT does, because it talks. Binding consciousness, language, and intelligence too tightly courts both false positives (seeing minds in ‘statistical acrobatics’) and false negatives (missing minds in animals or brain organoids). O’Connor adds a sharp diagnostic drawn from his own idealist leanings: whatever quality you invoke to call an AI conscious, strip it from a human — take away sight, hearing, memory — and if they remain conscious, that quality is no sufficient test.

Conceivability, metaphor, and the real problem

The pair turn conceivability against itself. A child can picture a plane flying backwards; someone who understands aerodynamics cannot — knowledge closes off what ignorance leaves open. Seth applies this to disembodied minds and silicon neurons alike: the more you know about brains, the less conceivable their substrate-independence becomes. He warns against the recurring habit, traced through Matthew Cobb’s The Idea of the Brain, of seizing the era’s most complex technology as the brain’s metaphor and mistaking the map for the territory — Whitehead’s fallacy of misplaced concreteness. His own stance is a lightly held ‘pragmatic materialism’ and a ‘real problem’ approach: not solving the hard problem head-on but explaining the properties of experience and their dependence on a substrate, betting that a materialist research programme will make more progress than its rivals, while leaving the ultimate question — what anything is — frankly open.

Speakers

• Anil Seth — guest; cognitive and computational neuroscientist; author of Being You and the essay The Mythology of Conscious AI
• Alex O'Connor — host; Within Reason / CosmicSkeptic

See also

• Keith Frankish on Illusionism, Consciousness, and the Hard Problem — the other Within Reason consciousness conversation; Frankish deflates qualia where Seth defends phenomenal realism grounded in biology
• Consciousness — the wiki map of positions; Seth supplies the positive mechanistic account the biological-naturalism entry lacks
• Illusionism — the deflationary rival Seth rejects: he is an avowed phenomenal realist
• Panpsychism — the expansionist rival; Seth’s exchanges with Philip Goff mark the contrast