Anil Seth on Consciousness, Machine Sentience, and Why Intelligence Isn't Awareness

Anil Seth

Both resist precise consensus definitions, but intuitively they are fundamentally different kinds of things. Intelligence is about doing something — solving a complex problem, or any kind of problem. In general it's about the capability to do things, functionally, behaviourally. Consciousness is primarily about feeling and being rather than doing. In the words of Thomas Nagel, more than fifty years ago — 1976, I think — for a conscious organism, for a conscious anything, there is something it feels like to be that organism. He doesn't mean it has to be emotional. Simply the idea that there's some interiority, some kind of experiencing going on. It feels like something to be me, to be you, but probably not to be a chair. This distinction between feeling and doing is intuitively right. It doesn't mean they're unrelated. In living creatures they're certainly related. But they may not have to go together in general.

Anil Seth

The overall framing is to reason through what has become a confusing topic: whether artificial intelligence could be conscious. It seems intelligent, at least in some ways. But there are a lot of somewhat unexamined views about whether AI can be not only smart but also conscious. Part of the discussion is to examine our own psychological biases — why we might even think an AI system could feel as well as do intelligent-looking things. Just because things go together in us doesn't mean they have to go together in general. Across living creatures there is a strong relationship: for certain kinds of conscious experience you need certain cognitive capabilities. If you're going to have the experience of regret, you need the cognitive competence to envisage alternative courses of action and their outcomes. Without that functional capability you can only experience something like disappointment or sadness. So the relationship goes that way. But it's not clear that being intelligent in some way has to be accompanied by consciousness.

Anil Seth

Let me add one nuance. The idea of AI being conscious isn't just that it results from some mechanism. It's usually the more specific claim that the mechanism is implementing computations, and it's in virtue of those computations that consciousness happens. That's the philosophical assumption of computational functionalism, rather than functionalism in general. Functionalism in general is quite a liberal idea — that consciousness is a property of the functional organisation of a system, an embodied embedded brain, which can encompass its internal structure and causal architecture. Computational functionalism is a subset that says it's the computations that matter. And when people think about computation they usually mean it in the sense inspired by Alan Turing: computations are independent of the material, independent of the substrate. There's a sharp division between software and hardware. The algorithm is what matters, and the algorithm is a mapping from one sequence of symbols to another. That's a strong claim, and the more you examine it the more it looks like it's on shaky ground.

Anil Seth

One way to examine it is the neural replacement scenario. This is put on the table when people want to justify the idea that consciousness could be implemented in silicon just as readily as in carbon. The thought experiment — David Chalmers popularised it, though it has earlier ancestry — is that I take one neuron and replace it with a perfectly functionally equivalent silicon alternative. It has to be perfect for the argument to carry through, because otherwise you can immediately say those differences may matter. I can do it for one neuron; neurons fire spikes, why not. Almost certainly nothing happens to your consciousness. If it doesn't matter for ten, why should it matter for any number — therefore substrate independence, therefore what matters is the functional organisation.

Anil Seth

But it begs the question. Ned Block, another philosopher, pointed out that maybe if consciousness depended on the actual stuff, then as you replace more and more neurons you'd experience consciousness fading — and you might not even notice as the subject. There's a more fundamental problem, though: like many thought experiments of this kind, the more you look at it, the less conceivable it is. It's really unlikely that you can perfectly replace a biological neuron with a silicon alternative. You can probably only replace it with another neuron — probably the same one. One example I like: some neurons fire spikes to clear the waste products of metabolism. How do you replicate that in silicon? You'd have to give the silicon a metabolism too, and that's not the kind of thing silicon can have.

Anil Seth

That thought experiment underpins a fundamental difference between brains and Turing-based digital computers: in brains you just can't separate what they are from what they do. That separation is enshrined in how we think about the computers we're familiar with — the ones running the server farms that underpin all the language models we talk to.

Anil Seth

Which is reasonable. There's a whole branch of synthetic biology where people create brain organoids — clusters of neurons, usually derived from human stem cells. They're human neurons; they wire up and start to show activity. They don't do anything very interesting yet, but they're made of the same stuff. We can't say for sure that's a necessary condition, but if you're creating something literally made of the same stuff, the whole uncertainty goes away. You no longer face the question of whether you could do it in silicon, because you're not doing it in silicon anymore.

Anil Seth

I'm very sceptical, but in discussions about what could be conscious we need a certain residual humility, simply because we don't know. I'm a phenomenal realist — I think conscious experiences exist. From that starting point, there's something about the embodied embedded biological brain that is sufficient for consciousness, and the game is to figure out what that is. Computational functionalism makes one specific bet: that below the level of the algorithmic description, things don't matter. But there are many other things that might matter, much more tied to a biological substrate. Anyone who says AI is conscious, or that it's impossible, is overstating what can reasonably be said. Still, I think digital silicon AI being conscious is very unlikely — partly because there are many reasons we over-attribute consciousness to these things, reasons that say more about our own cognitive biases, and more fundamentally because of this deep way you can't disentangle what brains are from what they do. Some people call it generative entrenchment. Put those together and there's good reason to be very sceptical that AI as we know it is capable of consciousness.

Anil Seth

I don't mean it in a metaphysically loaded sense. I'm happy to think of what something does as related to what it is, and to explain what it is in terms of what it does at lower levels. It's just that the more you do that, the less flexible you become about realising those same functions in different materials. Take something uncontroversial like building a bridge across a river. Only certain materials are up to the job. You can't build a bridge out of string, or cream cheese, or water. Some things have properties that depend on the material they're made of, and biological systems may be subject to many such constraints. That's not a radically controversial claim — I'm not saying there's something magical about carbon.

Anil Seth

There are weak and strong versions of 'the stuff matters'. The weak version: what matters is what it does, but what it does is only realisable in certain materials. The computational view pushes back — the whole point of computation is that you don't worry about what it is, many things can do it. But that has to be justified; you can't take it for granted as a starting point. The stronger version, which I'm attracted to but not sure is right, is that the stuff matters in a more fundamental way — similar to how the molecular structure of H2O matters for the properties of water. The factor that explains why there's phenomenality at all may be something like the energetics of metabolism, the self-producing autopoietic character of living systems. I don't know if that's true, but it's at least as plausible to me as the idea that it's a matter of computation.

Anil Seth

Exactly. Then you're talking about something different. So you'll need some understanding of what it means to be biological, to be a living system — and that only goes down to a certain level of complexity, because deeper than that you're just doing atomic physics.

Anil Seth

This is the really exciting open question, and it's the third part of the general case against conscious AI as we know it. The first is our collection of biases. The second is scepticism that substrate-independent computation is sufficient. The third, equally important and more challenging, is the view called biological naturalism — that consciousness is a property of living existence. The first expressions of that were a little blunt: almost an identity, with no positive reasons. I'm not satisfied with that. You've got to say what it is about biological systems that matters for phenomenality.

Anil Seth

There are no knockdown arguments, but one story runs from thinking about the brain as a prediction machine, in which the contents of conscious experience are underpinned by brain-based predictions about the causes of sensory signals. I've called this the controlled hallucination. That can be thought about computationally, because it's all about Bayesian inference, but it doesn't have to be implemented computationally — I don't think it is. The story tries to draw a line from that process, which tells us why conscious experiences have the characters they do, all the way down to metabolism and autopoiesis, where living systems by their nature have to maintain and produce themselves, regenerating their own material basis, keeping themselves out of thermodynamic equilibrium with their environment. If you're in thermodynamic equilibrium, you're no longer alive. And that too can be understood as minimising the surprise of sensory data, through the free-energy principle from Karl Friston.

Anil Seth

There's a lot in there, and not all the steps are clear. The trickiest is going from free energy in the thermodynamic sense — real energy, work being done to transform substrates so a cell can reproduce its parts — to the information-theoretic free energy that corresponds to prediction error when we think of the brain as a prediction machine. You can line these up, but I'm personally not yet satisfied with how it works.

Anil Seth

That's part of the mythology — some motivated reasoning is going on. We're still very human-exceptionalist. The march of science and philosophy has been repeated attempts to challenge that exceptionalism: we're not at the centre of the universe, not unrelated to other animals. Language has been one of the things we think sets us apart, and it's under pressure from LLMs — it'd be unfair to say they have no form of language — and from AI applications decoding non-human animal communication. You mentioned chess programs. When DeepMind's AlphaGo played move 37 I don't remember people thinking it was conscious, though there may have been a temptation to project some uncanny mind. The cleaner example is AlphaFold, another DeepMind system that predicts protein structures. I've never heard anyone worry that it's experiencing things while doing its protein-folding acrobatics. But under the hood it's very similar to an LLM — neural networks and transformers on silicon, tuned differently, trained on different data. If somebody thinks GPT is conscious but AlphaFold isn't, I'd be interested in what justifies the distinction. It's hard to justify.

Anil Seth

That's right. If we're too tempted to see things through the lens of what a conscious human is like, and bind consciousness, language and intelligence too closely, it leads to both false positives and false negatives. We can see consciousness where it's not there, in the statistical acrobatics of LLMs, and miss it where it really is — as we've done throughout the history of human treatment of non-human animals, and as we might now do with synthetic biology and organoids.

Anil Seth

That's a good argument. You're right that if you can do that, you certainly can't treat the presence of that quality as a sufficient condition. There might be interesting things to say about jointly sufficient conditions, but it's a good question to ask oneself. Unlike neural replacement thought experiments, I quite like this one, because stripping things away helps us get out of our human-centred perspective. There's an old version of this — Avicenna's flying man.

Anil Seth

Where I land, when I try to imagine what would be left, is something like the feeling of being alive. It's a bit unclear, but it's what I imagine might be there when you strip away all exteroceptive perception, all visual and auditory content, all cognitive content — not the Cartesian rational thinking. There's overlap here with states of pure awareness in meditation and some psychedelics. Strip away emotion, the experience of the body as an object in the world, the first-person perspective, memory, intention. Is there anything left? My intuition is this basic feeling of being a living organism, without shape or boundary or specific content, but persisting in time.

Anil Seth

Conceptual imagination is too low a bar. I can create a concept of a disembodied consciousness, write stories about it, but you can't use that to carry much weight. It's a problem with conceivability arguments in general. Can I imagine a Boeing 787 flying backwards through the air? Of course — I can picture it. But the more I know about aerodynamics, then given the laws of physics as they are, you can't have planes that fly backwards. So conceptually I can imagine these things, but if I want to understand how things are in the world as it is, then the more I know about the brain and body and how intimate and explanatory these connections are, the less it means to say I can conceive it.

Anil Seth

The more you know, the less likely it becomes that these things are separable. Same with neural replacement: if you know nothing about neurons, it's easy to imagine replacing one with silicon — especially if you've reified the metaphor of the brain as a computer. But the more you know about how real neurons work, how their collective behaviour self-organises and changes over time, the less conceivable it becomes, just like the plane flying backwards.

Anil Seth

It's already happening a little — even with AI, the metaphor is shifting towards distributed, self-organising systems like the internet. Matthew's book makes the point that we always have to be careful not to confuse the metaphor with the thing itself — this is Whitehead's fallacy of misplaced concreteness. The brain-as-computer metaphor is particularly tricky, because it also drags in theorems from computer science and physics about the generality of computation. There's an ongoing debate about what physical processes count as computation — John Wheeler thought everything is fundamentally computation.

Anil Seth

The temptation to think computation is all there is rests on two events. The first is Turing, who came up with the idea of an algorithm — mapping sequences of symbols to other sequences through a series of steps, like a mathematical recipe — and showed it was powerful with the universal Turing machine, an abstract machine that could implement any algorithm. But Turing's classic paper, now about ninety years old, was motivated by the halting problem — showing there are functions that can't be solved algorithmically. So there are limits even there. And many things in the real world aren't algorithmic — continuous things, inherently random things. There's more to the universe than Turing computation.

Anil Seth

The second event was Walter Pitts and Warren McCulloch, at the beginning of AI and cybernetics, who showed that simple neural networks — computational abstractions of neurons that sum their inputs and generate an output — could implement logical operations, and given storage could be Turing-complete. Put those together and you understand that if you idealise the brain as a network of neurons summing inputs and passing on outputs, you get everything computation can give you. That's powerful, and it's why for decades people thought we don't have to worry about anything else in the brain. But Turing computation can't do everything, and it's unclear it can realise all the properties of the brain. When you look at a brain, you realise you can't separate what it is from what it does, and that puts real pressure on the idea that Turing computation is all that matters.

Anil Seth

It's partly theoretical, partly practical engineering. Computers are useful because they're built this way: you can run the same program on different computers and get the same thing, or different programs on the same computer. That interoperability emerges from Turing computation being substrate-independent — once you define the algorithm, that's all you need to know what the system will do. But evolution didn't design brains for interoperability. The functional dynamics of my brain won't work in your brain, and there was never evolutionary pressure for that. In fact there's pressure the opposite way, because maintaining a sharp separation of scales — insulating one level of dynamics from another — is energetically very expensive. It takes energy to make sure ones stay ones and zeros stay zeros; that's one reason AI systems are so much more energy-inefficient than real brains. If you want energy efficiency, which evolution cares about, you get systems where what it does is much more entangled with what it is. And there may even be functional benefits to being scale-integrated in this way — brains seem to sit in a subcritical regime where small differences can make differences at larger scales and vice versa.

Anil Seth

This has been one of the workhorses of empirical consciousness research. The idea is to match as closely as possible the signals coming into the brain, but create conditions where in one case there's a conscious experience and in the other there isn't. A classic example is masking: you show an image, but if you surround it with other images in the right way, people don't consciously perceive the middle one, even though it's presented to their visual systems. In our lab we use continuous flash suppression — we show an image to one eye and a continually changing Mondrian of coloured squares to the other, and the person consciously sees only the changing squares. So you create a situation where sensory information is present and makes some difference, yet they don't report perceiving it.

Anil Seth

You diagnose the problem beautifully — it's a really tricky area, and there's still lively debate about whether anything counts as unconscious perception. Verbal report is key: if you verbally ask a split-brain patient whether they saw something they might say no, but if you ask them to draw it or push a button you get a different response. There's also a conceptual issue: a conscious experience might still be happening in your visual cortex while you have no cognitive access to it. Ned Block argues for this with his distinction between phenomenal consciousness and access consciousness — the idea that cognitive access is optional. That's very hard to get an experimental handle on, because if you don't have access, how would you ever know it was going on?

Anil Seth

It's not a silly question, but it's very hard to answer. There's a lot of debate about the unity of consciousness, and the split-brain case is a key pressure point. Some treat unity as axiomatic — there can only be one, like Highlander — because it's hard to imagine two disunified consciousnesses. The evidence with split-brain patients is really mixed; it gets interesting in detail. There seem to be problems integrating things across the midline rather than detecting them at all. I try to stay non-committal: consciousness seems unified to most of us most of the time, but I wouldn't build a theory requiring that always to be so. There might be cases where conscious experience becomes interestingly distributed — octopuses are a good provocation, where consciousness may be decentralised in a way it isn't for highly cephalised animals like us, whose nervous system is concentrated in the head. An octopus is a cephalopod — literally 'brain foot' — with more neurons in its limbs than in its central brain.

Anil Seth

Unity doesn't imply lack of differentiation. Most uses of the term suggest all these things happen within a unified conscious field. You can attend to different parts with different characteristics, but there's no way in which your consciousness of redness is happening completely separately from the rest of your visual field. It's all bound into one, though it has structure.

Anil Seth

A classic example, going back to Michael Gazzaniga: you show something prosaic to the left, verbal hemisphere and something funny to the right hemisphere, and the person starts laughing. Ask why, and they confabulate a reason based on what the left hemisphere saw, even though the real cause was shown to the other half. But the most amazing thing is that under almost any ordinary circumstance it's impossible to tell anything has happened — you need very specific experimental setups. And a lot of these effects resolve over time; very few operations completely segregate the hemispheres, and they're rarely done now because epilepsy medication has improved. A former postdoc of mine showed that in a split-brain patient each hemisphere could detect visual stimuli anywhere in the field — what they couldn't do was integrate across the midline. Tim Bayne suggests it may not be a stable thing: under some circumstances one agent, under others two. That puts pressure on the assumption that the number of conscious agents in a skull has to stay the same.

Anil Seth

There's at least one case of conjoined twins who share a brain and can't be surgically separated — I think they're in their teens now. They seem to be individuals with somewhat different personalities, but they share things: one can taste what the other eats. They're extraordinarily synchronised, and from videos they seem to complete each other's sentences, which might indicate they share at least the contours of a single thought. Though of course they're getting almost the same sensory input all the time, throughout their whole lives.

Anil Seth

I think it's both, and it would be strange if it didn't have that functional role. Attention is only slightly less mysterious than consciousness, but it is less mysterious, less metaphysically loaded. It's one of those things that seems natural until you ask what it really means. It affects conscious experience — we're more likely to be conscious of what we attend to — though I don't think it's a complete gate; the extreme view that we're only conscious of what we attend to is probably overstating it. Using the prediction-machine framing, attention isn't one thing but many ways of adjusting the gain, the signal-to-noise ratio — how much the brain updates its inferences based on some sensory data. Paying attention to something means this sensory prediction error is going to have more impact on the perceptual inference, on the continuing process of prediction-error minimisation.

Anil Seth

This is a great observation — David Chalmers has a nice piece, I think called 'What We Talk To When We Talk To a Large Language Model'. On one hand it highlights the trouble we get into if we're captured by the superficial fact that these things are linguistically competent. Under the hood they're massively different. It's not only silicon and algorithms — is there an identity? What is talking to you? It's not the foundation model, which is stable. Is it an instance, a server farm? A query could be processed half in Arizona and half in New York, the other way around next time. And you can leave a conversation for a day, come back, and from that conversation's seeming perspective it makes no difference — yet time is fundamental both to how biological brains work and to the nature of conscious experience. Noting these differences should highlight the danger of inferring AI consciousness from fluency with language. On the other hand, it's valuable to temporarily set aside whether they are conscious and ask what it would be like if they were, because that lets us probe issues like the unity of consciousness that are so hard to get out from under in the human case. It gives us insight into a wider space of possible minds — minds that need not be conscious, but that aren't constrained the way biological minds are, having evolved within bodies.

Anil Seth

I'm very pragmatic. When asked, I describe myself as a pragmatic materialist. I don't know that materialism is true, and I don't want to be out there defending it against all comers, but I've always found it a useful perspective because it has the resources to chip away at the problem of consciousness. That's the real-problem approach: consciousness exists, it has many properties — vision differs from emotion differs from memory — and I want a unified way of thinking about different kinds of experience that explains their relation, their character, and their dependence on a particular kind of substrate.

Anil Seth

The controlled hallucination view is an expression of this. It's a gloss on an old idea, perception as inference, going back in philosophy to Kant — the noumenon and the phenomenon — and in psychology to Hermann von Helmholtz, the first to describe perception as unconscious inference. The brain is trying to infer the causes of sensory signals, and that goes on under the hood. What we're aware of is the brain's best guess — its most likely interpretation of the causes of the sensory signals it gets. It flips the textbook view on its head. Classically, perception is reading out the world: information comes into the eyes and the brain reads it out. This view goes the other way: the content of experience, even though it seems to be out there and passively registered, is actively generated, and the sensory signals calibrate these predictions so they're useful for the organism. The checkerboard illusion is a classic example — two squares that look different colours but are the same. Even knowing that, you see them as different, because your brain constructs an interpretation that makes sense narratively rather than reading out the actual pixel value.

Anil Seth

Much more the latter. I don't think it tells us what consciousness is. I have an ongoing discussion with panpsychist philosophers like Philip Goff, who always says that everything we materialist-oriented scientists come up with about the brain is completely compatible with panpsychism, probably with idealism too. I think that's probably true. But the question for me is pragmatic. I don't know what things really are — you take a chair apart and find molecules, then atoms, then quarks, then quantum fields. Whether what actually exists is mental states, or stuff with qualitative properties as well as physical ones, I'm not qualified to answer. But a pragmatic materialism that says brains are made of stuff — and I'm more interested than most in what they're made of, not treating them as computers that happen to be made of meat — lets me ask whether I'm moving the needle on explaining the properties of the phenomenon, or barking up the wrong tree.

Anil Seth

If I were being unkind to myself, it's kicking the can down the road. But I don't think it's quite that, because the road changes and the can changes as you go. What we think of as the mystery of consciousness is a bunch of related mysteries — free will, and so on. The more we dissolve those or wrap them into a single framework, the more progress we make. Take the imperfect historical analogy of life: it was once thought to be beyond physics and chemistry, maybe even requiring something supernatural. We don't understand everything about life, but that sense of mystery has receded — much of what we thought definitional of life can be accounted for through biochemistry and physics. Life isn't the same as consciousness, which seems to have a different mode of existence. But aspects of it are describable. The sense that there was something incredibly resistant to explanation didn't persist over time, and I'm suspicious of the suspicion that it won't work for consciousness. I want to wait and see what's left. Maybe there'll still be a deep residual mystery; maybe there won't.

Anil Seth

And back to AI: we need a way of thinking about whether AI is conscious, because it underpins moral and ethical decisions and is already affecting policy and behaviour. From a lightly held materialist view, we start from the assumption that something about the materiality of an embodied embedded human brain matters. Maybe it's the computation — I doubt it — maybe something else. That gives us a research agenda: what aspect of our materiality matters, and why. Justified, testable answers will help us say what we need to say about AI, but also about non-human animals and all kinds of situations. A science of consciousness needs to address these immediate questions as well as the ultimate one of what consciousness really is.

Anil Seth

Thank you, Alex. It's been a pleasure.