Notes — Anil Seth on Consciousness, Machine Sentience, and Why Intelligence Isn’t Awareness

Notes on Anil Seth in conversation with Alex O'Connor — Within Reason, April 2026. Built around Seth’s prize-winning essay The Mythology of Conscious AI and his book Being You.

Four questions [Adler frame]

The four questions from Mortimer Adler’s How to Read a Book: what is it about, how is it argued, is it true, what of it?

Q1 — What is it about as a whole? Whether an artificial system could be not merely intelligent but conscious — and why Seth thinks the confident ‘yes’ rests on a category error. The spine is a single distinction: intelligence is doing (solving problems, acting), consciousness is feeling and being. The two co-occur in living creatures, but that is a contingent fact about us, not a law binding every possible mind. Seth then builds a positive, if unfinished, case that consciousness is a property of living systems, which no present-day computer is.

Q2 — How is it argued? Three moves stacked. First, diagnose the biases that make us over-attribute experience to fluent machines. Second, deny that the pro-AI premise — computational functionalism — is a safe starting point; it is a strong bet that has to be earned. Third, sketch a biological alternative running from the brain-as-prediction-machine down to metabolism, bridged (tentatively) by Friston’s free-energy principle. Throughout, Seth reasons by dismantling thought experiments from the inside — showing that the neuron-replacement scenario, examined closely, quietly destroys its own premise.

Q3 — Is it true, in whole or part? The feeling/doing distinction is clean and does real work; the biases (human exceptionalism, the seduction of language) are well-observed — nobody worries that AlphaFold suffers, though it runs on near-identical machinery to a chatbot. The generative-entrenchment objection to neuron-replacement is the strongest part: it is genuinely hard to specify a silicon neuron that clears its own metabolic waste without simply being a neuron. Weakest, and Seth flags this himself, is the bridge: joining information-theoretic free energy (prediction error) to thermodynamic free energy (the energetics of metabolism) is admitted as unfinished — ‘I’m personally not satisfied yet with how this works.’ Two pushbacks matter. Ned Block’s fading qualia reply is that experience might simply dim as the substrate changes, so the replacement argument begs the question — Seth accepts the point and leans on a different objection (inconceivability) rather than answering it. And a phenomenal realist like Seth still owes the deflationist (see Illusionism) an account of why there is anything to explain; he grants panpsychists like Goff that his findings are compatible with their metaphysics too, so the biological story is a research bet, not a proof.

Q4 — What of it? The verdict is calibrated scepticism, not a slogan: anyone who calls conscious AI either certain or impossible overstates the evidence. But the practical payoff is real — attributions of machine consciousness are already shaping policy and moral intuition, and Seth’s frame supplies a discipline for resisting the false positive (a system that talks) while staying alert to the false negative (animals, brain organoids). The stripping-away test is portable: any quality you would cite to call a machine conscious, remove it from a human — if they stay conscious, that quality is no sufficient sign.

Glossary

Computational functionalism — the claim that consciousness arises in virtue of computations, i.e. of abstract symbol-to-symbol mappings, and so is indifferent to what runs them. A narrower, stronger cousin of plain functionalism (consciousness as a property of a system’s functional organisation, which may still be tied to biology). The pro-conscious-AI position depends on this narrower bet.
Substrate-independence — the corollary that, once the algorithm is fixed, the material implementing it does not matter (the same program runs on any adequate hardware). True of computers by design; Seth’s central doubt is whether it is true of brains.
Generative entrenchment — Seth’s name for the way a brain’s what-it-is and what-it-does cannot be prised apart. Evolution never insulated one scale of dynamics from another — such insulation is energetically expensive — so the low-level material and the high-level function are mutually dependent, not stacked as separable software-on-hardware.
Controlled hallucination — Seth’s gloss on perception as inference: what we experience is not a read-out of the world but the brain’s best guess at the hidden causes of its sensory signals, continually calibrated against those signals. Perception runs outside-in in appearance but inside-out in fact.
Free-energy principle — Karl Friston’s proposal that a self-maintaining system acts to minimise ‘surprise’ (roughly, prediction error) about its sensory states. Seth uses it as the candidate bridge from prediction to biology, while conceding the two senses of ‘free energy’ — information-theoretic and thermodynamic — are not yet cleanly joined.
Autopoiesis — literally self-producing: the mark of a living system, which continually regenerates its own material parts and holds itself out of thermodynamic equilibrium with its environment. Equilibrium, for a living thing, is death.
Biological naturalism — the position that consciousness is a property of living existence. Seth adopts a refined version: dissatisfied with the bald identity (‘consciousness is just what living systems have’), he wants positive reasons — what about being alive matters for experience.
The ‘real problem’ vs the hard problem — the hard problem (Chalmers) asks why there is any experience at all attached to physical processing. Seth’s real problem sidesteps a frontal assault: explain the properties of particular experiences and their dependence on a substrate, betting that the deep mystery recedes as the explanations accumulate — as the once-‘magic’ mystery of life did.
Phenomenal realism — the view that conscious experiences genuinely exist and are what they seem — there really is something it is like. Seth holds it firmly, which sets him against the illusionist (see Illusionism) who treats the seeming as itself the thing to explain away.

Intelligence is doing; consciousness is feeling

The essay exists to protect one distinction. Intelligence is a capacity to do — to solve problems, to act to some end. Consciousness, in Nagel’s sense, is that there is something it is like to be the system: an interiority, an experiencing, present in you and me and probably absent in a chair. The two are related but not identical. Some experiences presuppose cognitive machinery — you cannot feel regret without the competence to imagine roads not taken; lacking that, the most you can feel is disappointment. But the entailment runs one way only. Being cognitively capable does not, on any argument Seth can find, force experience to accompany the capability. The whole conscious-AI debate, he holds, smuggles in the reverse entailment: it sees a system doing intelligent-looking things and infers that it must therefore feel. O’Connor names the source of the error — we reason from the single case we have, ourselves, where doing and feeling coincide, and over-generalise the pairing to all possible minds.

Computational functionalism is a bet, not a floor

The engine of the pro-AI view is not merely that consciousness is mechanical but that it happens because of computations — Turing-style mappings of symbol-strings to symbol-strings, defined independently of any implementing stuff. That is a strong and specific claim, and Seth’s key methodological point is that it cannot be assumed as a starting premise; the burden is on its holder to show that computation suffices. Two limits press on it. Turing computation cannot capture everything — continuous and genuinely random processes escape the symbol-mapping model, and even within computer science some functions are not algorithmically solvable (the halting problem). So it is far from obvious that Turing computation can realise every property a brain has. The weaker, more defensible cousin — consciousness depends on functional organisation, but that organisation may only be realisable in some materials — Seth is content with. It is the substrate-independent version, the one that licenses conscious chatbots, that he thinks stands on shaky ground.

The neuron-replacement critique

The stock argument for silicon consciousness: swap one neuron for a functionally perfect silicon equivalent — nothing changes; do it for ten, a million, all 86 billion — where would consciousness go? Therefore substrate does not matter. Seth grants the set-up, then attacks the premise rather than the inference. Two objections, of unequal weight. The first is Block’s: the argument begs the question, because experience might simply fade as neurons are replaced, unnoticed by a subject whose capacity to notice is itself fading. The second, which Seth prefers, is that the premise is barely conceivable. A ‘functionally perfect’ replacement has to reproduce everything the neuron does — and some neurons fire spikes to clear the waste products of metabolism. To match that in silicon you must give the silicon a metabolism; push the fidelity further and you descend past anything recognisably silicon to a particular arrangement of electrons and protons — at which point you have not engineered a computer, you have built a neuron. The bridge-and-cream-cheese analogy carries the intuition: a bridge needs not just the right structure but the right material — you cannot span a river with cream cheese, and replacing each girder atom-for-atom just gives you back steel. What looks like manufacturing a conscious machine is really assembling a ‘Frankensteinian biological being’ and calling it artificial because you made it by hand. Brain organoids — clusters of real human neurons grown in a dish — make the point from the other side: build the thing out of the same stuff and the whole silicon-or-not uncertainty simply dissolves.

The biological bridge Seth is not yet satisfied by

Pressed on how strong his conclusion is, Seth is careful: very sceptical, but reserving ‘a certain residual humility’, because we do not know. His positive story is a refined biological naturalism. He rejects the blunt early version — consciousness is just a property of living systems, full stop, no reasons given — and reaches for a mechanism. The line runs: the brain is a prediction machine, and the contents of experience are its best guesses at the causes of sensory signals (controlled hallucination); this can be described in the language of Bayesian inference without being implemented as computation — Seth thinks it is not computed in the brain but done in other ways; and the same predictive framing extends down to what it is to be alive — metabolism and autopoiesis, a system holding itself out of thermodynamic equilibrium, minimising the surprise of its own states via the free-energy principle. The load-bearing and admittedly unfinished step is the join between the two ‘free energies’ — the thermodynamic sort that metabolism trades in, and the information-theoretic sort that prediction error names. Seth offers the whole line not as proof but as at least as plausible as the computational bet — the point being that the biological wager is a live scientific competitor, not a mystical retreat.

Why LLMs fool us and AlphaFold does not

Seth locates the credulity in us, not the machines — three biases do the work. Human exceptionalism: we keep nominating some faculty as what sets us above nature and keep being corrected. The conflation of intelligence with consciousness: the unexamined pairing again. The seduction of language: a system that talks trips a deep intuition that a mind is present. The clinching comparison is AlphaFold, architecturally close kin to a large language model — networks and transformers on silicon — which never tempts anyone to worry about its inner life, because it folds proteins rather than chatting. If you hold that GPT is conscious but AlphaFold is not, what justifies the line? Binding consciousness, language and intelligence too tightly courts errors in both directions: false positives (seeing a mind in the ‘statistical acrobatics’ of an LLM) and false negatives (missing minds in animals, or in organoids). O’Connor supplies a sharp diagnostic from his own idealist leanings — the stripping-away test: take any quality you would invoke to call a machine conscious (sight, memory, humour) and remove it from a human; if the human stays conscious, that quality cannot be a sufficient sign of consciousness anywhere. Seth accepts it as a good test of sufficiency, adding only that jointly-sufficient bundles might still be interesting.

Conceivability is not knowledge

The pair turn the conceivability move against itself, and this is one of the note’s sharper threads. A child can picture a plane flying backwards; someone who understands aerodynamics cannot — real knowledge closes off what ignorance leaves open. Conceptual imagination is too low a bar to carry metaphysical weight: one can write a story about a disembodied mind, but story-telling proves nothing about what the laws of nature permit. Applied to the cases: the more you know about how real neurons self-organise, the less conceivable their silicon substitution becomes — inconceivability grows with understanding, exactly as with the backwards plane. Seth folds in a warning from Matthew Cobb’s The Idea of the Brain: in every era we seize the age’s most complex technology as the brain’s metaphor — hydraulics, telephone exchanges, now the computer — and then forget it is a metaphor, mistaking map for territory. This is Whitehead’s fallacy of misplaced concreteness. The computer metaphor is uniquely sticky because it drags in real theorems — Turing’s universal machine, McCulloch and Pitts showing that idealised neural nets can be Turing-complete — which together tempt us to conclude that everything else about the brain can be thrown away. It cannot: Turing computation does not do everything, and a brain is precisely the kind of thing whose what-it-is will not separate from its what-it-does. [?] The dating Seth gives in passing (Nagel ‘more than 50 years ago’, Turing’s paper ‘like 90 years old’) is offered loosely and should be treated as approximate.

The real problem and a lightly-held materialism

Seth’s own stance is a ‘pragmatic materialism’ worn lightly — not a frontline defence of materialism against all comers, but a working posture chosen because it has the resources to chip away. The real problem approach: take phenomenal realism as given (experiences exist and differ — vision unlike emotion unlike memory), then seek a unified account of their characters, their relations, and their dependence on a substrate. Controlled hallucination is one such unification — start from perception-as-inference and the same machinery illuminates emotion, the sense of free will, more. O’Connor’s music analogy frames the worry: you can build an entire theory of harmony without ever saying what sound is (vibrations in air) — is Seth’s programme a theory of consciousness that never touches what consciousness is? Seth half-concedes it looks like kicking the can down the road, but insists the road and the can both change as you walk: dissolving the cluster of related mysteries (free will among them) is progress, and the deep residual — ‘what actually exists’ — he leaves frankly open, granting the panpsychist Goff (see Panpsychism) that his neuroscience is compatible with panpsychism and idealism alike. His historical wager is life: once thought to need something beyond physics and chemistry, now largely accounted for without the mystery ever being ‘solved’ head-on. He is ‘suspicious of the suspicion’ that consciousness must be the permanent exception — and wants to strip away the answerable questions and see what, if anything, is genuinely left.

What are we even talking to? The identity problem for LLMs

A quieter but telling strand (Seth credits a Chalmers piece, What we talk to when we talk to a large language model [?]). Even granting, for argument’s sake, that an LLM were conscious, what would be the bearer? Not the stable foundation model; an instance? a server farm splitting your query across Arizona and New York, differently next time? Each of your conversations is a separate stream; you can leave one for a day and resume as though no time passed. Yet time is fundamental both to how biological brains work and to the character of experience. The mismatch is itself evidence: the fact that we cannot even frame what conscious AI would pick out suggests the question may be partly senseless as posed. Handled carefully, though, the same exercise is useful — LLMs, conjoined-twin cases, split-brain patients and octopuses (whose consciousness may be genuinely decentralised) all pressure the assumption that one skull hosts exactly one unified mind, and so map a wider space of possible minds — most of which need experience nothing at all.

Anil Seth on Consciousness, Machine Sentience, and Why Intelligence Isn't Awareness