Sean Carroll on Vacuum Energy, the Cosmological Constant Problem, and the Anthropic Principle

Sean Carroll

Physics is a very broad field — atomic physics, plasma physics, condensed matter physics, biophysics, particle physics, gravity, cosmology. Different fields have different rates of progress, both because of theoretical ideas and experimental input. If you focus just on fundamental physics — figuring out the most fundamental laws of nature, not including things like biophysics and condensed matter physics, not because they're not important, it's just a different thing — there haven't been many surprising experimental discoveries in the past several decades, arguably since the 1970s. There have been a few here and there, masses of neutrinos, things like that. Also things we expected, like the Higgs boson and gravitational waves. But in terms of true surprises, it's been few and far between. That makes it hard to make progress in fundamental physics, because experiments are what drive us to great ideas.

Sean Carroll

There is one shining counter-example: the acceleration of the universe. Back in 1998, two competing astrophysics groups looked at Type Ia supernovae, using them to measure the expansion rate of the universe and how it changed over time, and told us that, against all expectations, the universe is accelerating, not decelerating, in its expansion. It was a big deal — Nobel prizes were handed out. We still don't understand with complete confidence what is going on with the acceleration of the universe, which is great for theorists trying to figure out what's going on. It would be helpful to have more than that one fact, but it doesn't slow us down — we're going to invent theoretical models to explain it anyway. There is a leading candidate, the cosmological constant, as proposed by Einstein many years ago, but there are other ideas as well.

Sean Carroll

I fell behind on recording podcasts recently, so I'm filling in with some solo episodes. I asked my Patreon listeners for a topic, and the clear favourite was theories of dark energy and the accelerating universe. In some sense it's shocking I haven't done that already — this is something I know something about. My first ever attempt at writing a popular-level book, around 2006, was a proposal on the accelerating universe and dark energy, and no one wanted to publish it. People said, "1998 to 2006, that ship has sailed." I don't think that was right — there's plenty of room for a really good book on the topic — but I never did write it.

Sean Carroll

When I started sketching this episode out, I realised there's too much to say for one podcast, so I've broken it into two. There's a natural dividing line: you can talk about the cosmological constant as Einstein came up with it, and what it means in relation to quantum field theory and observation, and then you can go beyond that to ask what if it's not a cosmological constant — what if it's something dynamical, a modification of gravity, or something else. Those are the two solo episodes. In this one, I'm going to talk about the cosmological constant, the energy of the vacuum, because that's an equivalent idea — they're precisely identical. The next one will be about dynamical dark energy, which contains a whole bunch of exciting ideas but currently has no evidence behind it, so we can let our imaginations roam a bit more. I'm going to try, across both, to let you in on what the questions are and why certain answers look more promising than others, so you have a road map for what we're trying to do in cosmology and fundamental physics with this problem, and what might happen next.

Sean Carroll

When we're talking about the cosmological constant, the energy density of the vacuum, there's a very obvious starting point: Einstein, who came up with the idea. It's a fun excuse to think historically, because we have different pre-existing ideas than physicists had back then — different things we think are important, and things they cared about that we might not. That's very much the case for Einstein, who was absolutely not a shut-up-and-calculate kind of guy. He had very strong ideas about how things should be, and thought them through rigorously and carefully, but he was open to changing his mind when he realised he was wrong.

Sean Carroll

Around 1915, Einstein had put the finishing touches on the general theory of relativity. General relativity says that space-time — an idea that went back to Minkowski right after Einstein worked out special relativity in 1905 — has a geometry: it can be curved, and that curvature is what we experience as gravity. That's 1915. I can't imagine how much fun it must have been to be a physicist at that time, with this new playground to explore. Einstein had the head start on everybody else, so many of the fun ideas one could explore in general relativity were first explored by Einstein himself very soon after he invented the theory. Not everything — the Schwarzschild solution, the curvature of space-time for something like the solar system or a black hole, was found only two years later, but by Schwarzschild, not Einstein.

Sean Carroll

One thing Einstein did do was think about the universe as a whole. He was heavily influenced by Ernst Mach, a physicist and philosopher of physics in Germany at the time, who had very definite ideas about the relationship between space, time, and matter. I'd say Mach's influence on Einstein was more inspirational than substantive — it nudged Einstein in a certain direction, but the answers Einstein came up with are ultimately disconnected from Mach's ideas. People keep trying to fit Mach's principle, which connects the rest frame of the universe to the existence of matter in it, into general relativity, even though the two aren't really that closely connected. But Einstein thought they were, and that mattered when he first turned his attention to cosmology.

Sean Carroll

There was also a puzzle in the back of Einstein's mind from Newtonian gravity, one people mostly forget about today. You can solve Newton's equations for gravity using a formulation — due to Pierre-Simon Laplace — in terms of a gravitational potential rather than action at a distance: the sun sources the potential field, and the Earth moves in it. That makes perfect sense for the solar system, but it's puzzling for the universe as a whole, because if you imagine a universe filled with a constant density of matter, the potential is undefined — there are multiple solutions that work equally well. That didn't seem a good starting point for cosmology.

Sean Carroll

So Einstein, rather than assuming the universe was spatially flat — remember, the whole point of general relativity is that the geometry of space and time is what defines gravity — asked what if space is finite. That would help with the Newtonian puzzle, and might be consistent with Mach's principle, that it's the matter in the universe fixing our local standard of rest. As far as I can tell, Einstein was pretty convinced there were good philosophical reasons to think space is finite, not infinite. Personally, I'd rather be open-minded about that; I'll let the data, or a better theory, decide. But he tackled cosmology by assuming matter is distributed uniformly throughout the universe. He had no real evidence for that — there was some indication of a uniform distribution circa the 1910s, but we didn't even know about the existence of other galaxies yet; that data didn't come until the 1920s. It was open season for speculation.

Sean Carroll

Einstein imagined a spherical universe — space as a three-dimensional sphere, no problem for differential geometry — and time as a line running into the future. He plugged this into Einstein's equation and asked what it implied for the evolution of the universe: a constant density of matter in a spherical universe. What he found was that both the geometry of space and the existence of matter contribute to a change in the size of the universe, and you can find a solution where, at one moment, the universe isn't changing size — you balance the curvature of space against the matter. What you can't do is keep it there. The equations implied a universe that starts from somewhere, expands, reaches a point of zero expansion rate, and then starts collapsing again — a closed, finite-time universe. Doesn't seem to be the universe we live in, but Einstein wasn't sure at the time. So he could find a static solution, but it would instantly start expanding or contracting depending on how you set the numbers.

Sean Carroll

There's a somewhat mythical version of this story that says Einstein was just philosophically devoted to a static universe. As far as I can tell, that's not quite true. Closer to the truth is that Einstein asked his astronomer friends whether the universe was expanding, and they said no, it seems to be constant, more or less static — though they didn't know there were other galaxies; they were just looking at stars within our own. So Einstein said he'd listen to the data, and went back to find a different solution where the universe is neither expanding nor contracting. To do it, he changed his equation, adding a new term to the space-time geometry side.

Sean Carroll

These days you might think that was a bit of a panic move — changing your own equation right away. But it was Einstein's equation; it wasn't sacred to him, and it hadn't been tested through decades of hoops. He asked what he was allowed to add, and realised he could add a term proportional to the metric tensor — the object that tells you distances and times in the space-time geometry. There's nothing wrong with adding such a term; it's completely compatible with the symmetries and principles he'd already established. So he added a new constant of nature, which he called the cosmological term, and we call the cosmological constant. Very roughly speaking, if this number is positive, it resists the tendency of matter to pull the universe together — it pushes the universe apart. By cleverly choosing its value, Einstein found a solution where space was a three-dimensional sphere, matter was uniform, and a positive cosmological constant balanced the gravitational pull of matter to give a static universe. This exact solution is now known as the Einstein static universe.

Sean Carroll

Almost right away, people realised this wasn't a good idea. It's often said the Einstein static universe is unstable, which isn't quite accurate, though I know what people mean by it. You're balancing the density of matter against the cosmological constant, and you have to balance it exactly — any slight mismatch and the universe would eventually begin to expand or contract, even if it stayed nearly static for a long time. That's not quite an instability, just a pointer to how finely tuned the balance is, with no good reason to be that precise.

Sean Carroll

There's a better objection, which I don't know whether anyone raised at the time: the universe is not perfectly smooth. Einstein knew density differs between the Earth, the sun, and interstellar space, so the assumption of constant matter density is at best an approximation — there are inhomogeneities, and they generally grow with time. Overdense regions pull more matter onto themselves, which is exactly what we think does happen — it's the origin of structure in the universe. But in the modern picture, that growth has only been going on for a finite time, because there was a big bang roughly 14 billion years ago. Einstein's model was supposed to last for infinity; there was no big bang in it. So why hadn't all the structure already formed? Why wasn't the universe arbitrarily inhomogeneous? That was a question he wouldn't have been able to answer.

Sean Carroll

Those puzzles became less pressing about ten years later, when Edwin Hubble and others put forward data showing the universe is not static after all. Hubble did two things. First, he showed that the little fuzzy nebulae people were photographing through their telescopes are, in fact, whole separate galaxies, very far away. Then he compared the distances of those galaxies to their redshifts and showed the universe is expanding. He was cautious about interpreting the data in light of general relativity — he was an observer at heart, not a theorist — so he just said there's a relationship between redshift and distance. But theorists like Einstein, and also people like Lemaître and de Sitter, instantly knew what he was seeing was the expansion of space.

Sean Carroll

There's some controversy in the historical literature about whether Einstein really called the cosmological constant the greatest blunder of his life — for not simply predicting that the universe should be expanding or contracting, which might have made him even more famous. We don't really know; that story was apparently told by George Gamow, who liked telling stories, and it might well be true. Einstein certainly did say he should get rid of the cosmological constant once he realised the universe was expanding — the whole point of it had been to make the universe static, and if the universe is expanding, you don't need it for that.

Sean Carroll

The problem with that move is: who cares whether you think you need it any more? It's there as an open possibility — it was a perfectly legitimate addition to the equations. The scientific thing to do isn't to say "I don't need it, I'll set it aside," but to ask, "Is it there?" What would its effects be? Can we put observational limits on it, or even detect it? Ever since Einstein suggested it, there's been a cycle of people suggesting the cosmological constant could help with some particular problem in observational cosmology, and then realising they don't need it after all, and it waxes and wanes.

Sean Carroll

That was the situation, and cosmology at the time — the 1920s, even into the '30s, '40s, '50s, maybe '60s — was not the most respectable part of physics. The ratio of theory to data was quite large, so it was hard to make progress, much as it's hard to make progress right now in various questions in fundamental physics. So people moved on to other things. The really good physicists were thinking about quantum mechanics and particle physics.

Sean Carroll

It was fairly quickly realised — by the 1930s — by people like Georges Lemaître, the famous cosmologist and pioneer of the Big Bang theory, that you don't need to think of the cosmological constant as a change to Einstein's equation at all. You can just include it in the original 1915 equation. This is a point that's perfectly transparent but people sometimes fumble it, so it's worth getting exactly right. Einstein's equation has a left-hand side — symbols built from the curvature of space-time, R-mu-nu minus a half R g-mu-nu — and a right-hand side, 8-pi-G times T-mu-nu, where T-mu-nu is the stress-energy tensor, summing up all the forms of energy, momentum, pressure, and density that curve space-time. The left side says there's space-time curvature; the right side says there's matter and stuff; the equation sets them proportional.

Sean Carroll

When Einstein first invented the cosmological constant, he put it on the left-hand side, grouped in with the curvature terms — lambda times the metric tensor. What Lemaître realised is that you can just do the algebra: move that term from the left-hand side to the right-hand side, with a sign flip, and identify it as a new form of energy — the energy that exists even in empty space. So Lemaître said you can think of the cosmological constant as the energy density of the vacuum, the vacuum energy. It's constant by hypothesis, and there's a pressure associated with it, which turns out to be negative — more like a tension. The classic example is a rubber band: pull it, and it pulls back on you. That's negative pressure — pressure pushes, negative pressure pulls.

Sean Carroll

What I want to emphasise is that there is literally no difference between these two ideas. Some people still insist there is, but they're just wrong — it's not as if you're either changing the geometry side of Einstein's equation or adding vacuum energy. They're exactly the same thing. From that perspective, the cosmological constant term becomes even more inevitable-sounding: there's a constant of nature, the energy density of empty space, and you have to go measure it. Maybe you can set it to zero for convenience, as a simplifying assumption, but it's in principle something you should constrain with data.

Sean Carroll

This negative-pressure idea leads to a weird, common explanation for why the cosmological constant makes the universe accelerate, and I want to flag that I don't like this explanation, so don't trust it. It goes: ordinary matter and radiation have positive energy density and positive pressure, so they make the universe decelerate — all those galaxies pulling on each other slows the expansion. But the vacuum energy has negative pressure, and therefore it pushes things apart. And you should be thinking, wait a minute — you just told me negative pressure is tension, which pulls things together, not apart. Then you're told, ah, but what I mean is the gravitational effect of negative pressure is to push things apart, because the expansion of the universe responds not just to energy density, but to energy density plus three times the pressure. Which is true — and that number three is the number of dimensions of space, not arbitrary.

Sean Carroll

So rho plus three-p, where rho is the energy density and p is the pressure, is the quantity that comes into the acceleration equation in general relativity. For the cosmological constant, p equals minus rho, so rho plus three-p equals minus two rho — negative, if rho is positive, and that's what pushes the universe apart. The problem with that explanation is: where do you get any intuition for the claim that rho-plus-three-p is what appears on the right-hand side of the acceleration equation? It's just magic unless you've actually gone through the maths. You're not really explaining anything to anybody. I don't like that explanation.

Sean Carroll

There's another way of explaining it that I prefer. The equation cosmologists actually solve for the expansion rate of the universe — this is all post-Einstein; Robertson and Walker, then Friedmann and Lemaître and others, generalised Einstein's spherical model to all sorts of possibilities, arriving at what's now called the Friedmann equation. The one everyone solves sets the expansion rate — the Hubble parameter, named for Hubble, who measured it — proportional to the energy density of the universe, plus a term for spatial curvature if you want it, though these days we know curvature is close to zero, so we can drop it. So: H-squared, the Hubble constant squared, is proportional to the energy density. That's it — no pressure appears in that equation at all.

Sean Carroll

So how do you get the difference between accelerating and decelerating out of that equation? The answer is that it depends on how the energy density changes with time. Imagine a universe with nothing but vacuum energy — no matter to worry about. Then rho, the energy density, is just a constant. H squared is proportional to rho, so H itself is proportional to the square root of rho, and if rho is constant, so is H. The expansion rate of the universe is a constant.

Sean Carroll

You might now object: you told me this means the universe is accelerating, and now you're telling me the expansion rate is constant — that doesn't sound like accelerating. That's because you're thinking of the expansion rate as a velocity, and it isn't. In an expanding universe, a galaxy a billion light-years away is moving apart at some apparent velocity, and a galaxy twice as far away is moving apart twice as fast, by Hubble's law. Different velocities for different galaxies — so the expansion rate can't be a velocity. It's a rate, the answer to the question: how quickly does the universe double in size? The Hubble time, one over the Hubble constant, is almost exactly that doubling time. So H being constant means the doubling time is constant: the universe goes from its initial size, to twice that, to four times, to eight times, sixteen times, and so on — exponential expansion, because you keep multiplying by two at every time step. That is obviously accelerating.

Sean Carroll

That chain of logic is longer than the negative-pressure explanation, but it actually makes sense, unlike the negative-pressure business, which isn't wrong, exactly — it's just making it sound easier than it really is. Okay, so that's the state of play in the 1930s. Einstein had come up with his equations and the cosmological constant; Lemaître had interpreted it as the energy density of the vacuum. But Lemaître wasn't doing quantum mechanics — this is all still perfectly classical. Lemaître was just pointing out that this number Einstein had invented could be interpreted as the energy density of empty space: take a cubic centimetre of space, remove all the particles, all the radiation, all the dark matter, and ask how much energy is left. According to Einstein and Lemaître, that's a new constant of nature — not something you can just say should be zero because it's empty. It's something you have to go out and measure.

Sean Carroll

At this era, the 1930s onward, a lot of the excitement in physics was in quantum mechanics and particle physics. People knew and loved general relativity, but it wasn't front and centre in the minds of the leading theorists. Wolfgang Pauli wrote a whole textbook on general relativity when he was preposterously young, so people did know about it, but it wasn't what they were thinking about day to day — they were thinking about quantum field theory. And the idea of vacuum energy rears its head there too, in a different way — something people also get a little wrong, so let me try to explain it carefully.

Sean Carroll

Forget quantum field theory for a moment, and think about the simplest physical system there is: the simple harmonic oscillator — literally a weight on a spring, bouncing back and forth. We describe it with a potential energy that looks like omega-squared x-squared, where omega is the frequency of oscillation, plus ordinary kinetic energy. It's simple to write down, but it's beautiful, and it illustrates a lot of basic ideas. If you take a quantum mechanics class, one of the first things you do, and one of the things you keep doing at higher and higher levels of sophistication throughout your career, is study the simple harmonic oscillator.

Sean Carroll

Classically, studying the simple harmonic oscillator means a ball rolling in a hill shaped like a parabola: let it go at zero velocity at some point, solve its equations of motion, and you get sines and cosines. The quantum mechanical version means solving the Schrödinger equation in that same potential — a wave function instead of a rolling ball. What you get is a series of states with fixed, well-defined energy, the eigenstates of energy, and for the harmonic oscillator these are exactly, equally spaced: there's a lowest energy state, and a next state exactly one h-bar-omega above it, where h-bar is Planck's constant, and so on, equally spaced all the way up.

Sean Carroll

There's one thing about the lowest energy state that's a bit mysterious, and how much you dwell on it depends on your professor. Classically, the lowest energy state of the harmonic oscillator is the particle just sitting at x equals zero — zero potential energy, zero kinetic energy. That's how we're trained to think. Quantum mechanically, that turns out not to be right. The lowest energy state doesn't have zero energy — it has one-half h-bar-omega. This phenomenon — the ground state energy of the quantum oscillator being one-half h-bar-omega rather than zero — is called the zero-point energy of the harmonic oscillator.

Sean Carroll

I think the answer is you shouldn't fret too much about it, and the reason is that one-half h-bar-omega isn't the ground state energy of the simple harmonic oscillator — it's the difference between the ground state energy of a classical harmonic oscillator and a quantum mechanical one. Quantum mechanics raises the ground state energy by one-half h-bar-omega, but it doesn't tell you what it was to begin with. I could have started with a potential that was omega-squared x-squared minus one-half h-bar-omega, and nothing stops me from doing that. Classically, adding a constant to the overall energy makes no difference whatsoever to the motion of the particle — it has no effect on the physics. All that matters is how the energy changes from point to point.

Sean Carroll

So two things: the actual zero-point energy is completely arbitrary — I can set it to be whatever I want, it's not intrinsically one-half h-bar-omega. And that number isn't even anything intrinsic to nature; it's the difference between a quantum-mechanical theory and a classical one. But nature is just quantum mechanical — it doesn't know about the classical theory you started from. You should have just started with the quantum theory in the first place, where the relevant fact is that the energies of different states are separated by h-bar-omega. Nevertheless, people get hung up on this, and start thinking quantum mechanics tells you there is energy density in the vacuum. There could be — that's a constant of nature, a fact about reality — but the zero-point energy of the simple harmonic oscillator, specifically, is not a fact about your quantum mechanical calculations. You can choose it to be whatever you want.

Sean Carroll

So why do we care about the simple harmonic oscillator? Because we want to get into the messy real world of electrons and protons, which means quantum field theory — fields stretched throughout all space and time, which vibrate, and which you quantize, and that's what gives rise to particles. Since the 1930s, getting into high gear in the 1950s, quantum field theory has been our best attempt at understanding the world at a deep level, at least outside gravity.

Sean Carroll

When you start studying quantum field theory, just like other areas of physics, you start simple and work your way up. That means starting with a free field — one that isn't interacting with other fields, or with itself — an ancient move, going back to Galileo ignoring air resistance. The second move, always what you do when you study fields, is to start thinking about waves of definite wavelength, what we call modes of the field. Rather than thinking about what's happening at each point in space, you think about what's happening at each wavelength — this is Fourier analysis, and it's literally what your ears do when you hear pitch.

Sean Carroll

Miraculously, each mode of a free quantum field obeys an equation exactly the same as that of a simple harmonic oscillator. That's why I was telling you about the harmonic oscillator — free, non-interacting quantum fields naturally behave like an infinite collection of simple harmonic oscillators, one at every wave vector, not at every point in space. Every mode of the field has the same kind of zero-point energy contribution as the ordinary harmonic oscillator. And that means you don't just have a vacuum energy density of one-half h-bar-omega — you have the sum, or integral, over all possible omegas of one-half h-bar-omega. And that equals infinity. This is one of the famous infinities of quantum field theory, perhaps the most basic one: the energy of empty space is infinite, if you're very naive about it.

Sean Carroll

But before you get too excited about that, everything I said about the quantum mechanical oscillator is still true. I could choose the vacuum energy to be whatever I want — it doesn't need to be infinity, I can just choose it to be zero. This is one of the steps in what we call renormalization of the quantum field theory, and it's perfectly legitimate — nothing forces the vacuum energy to have a particular value to begin with. So the lesson isn't that there's an infinite amount of energy in empty space; the lesson is that there's an arbitrary amount, and that's exactly Lemaître's lesson about the vacuum energy all over again: there's a new constant of nature, and we have to go measure it. Quantum field theory says there can be vacuum energy in empty space — it does not say what it has to be. But there's another level, which lets you guess what it could naturally be.

Sean Carroll

The first person to go down that road was Yakov Zeldovich, a Russian physicist who, in the 1960s, made clear that the vacuum energy of quantum field theory is precisely what feeds into the vacuum energy of relativity — the cosmological constant. That's the beginning of the connection between quantum field theory and the cosmological constant. There were precursors before Zeldovich, and it's a fascinating story, because people didn't quite know what was going on, so they clung to certain beliefs and let go of others without always putting them in the right context.

Sean Carroll

Richard Feynman became famous for Feynman diagrams in quantum field theory, derived one hundred per cent from quantizing fields, not from thinking about particles as fundamental. But before that, Feynman had an earlier idea with his advisor John Wheeler at Princeton — the Wheeler-Feynman absorber theory of radiation, part of the package where Wheeler proposed that positrons might just be electrons moving backward in time, so there's only one electron in the universe, moving forward and backward through it. It's not right — it was a good guess, a fun idea, but we now know quantum field theory is what's actually right: all electrons are the same not because they're literally the same electron, but because they're all vibrations in the same underlying field. But at the time, Wheeler and Feynman were trying to think of electrons as particles rather than fields, and one of the things in the back of their minds was that quantizing field theory gives an infinite contribution to the vacuum energy. They worried that if they could replace field theory with a theory of particles, that problem would go away. It didn't work, but it was part of their motivation, and it eventually led to Feynman diagrams, which is pretty good.

Sean Carroll

The other example is Phil Anderson, a Nobel Prize-winning condensed matter physicist who was arguably the first person to propose what we now call the Higgs mechanism — he thinks it should have been called the Anderson mechanism. The idea: imagine a scalar field, just a number at every point in space, that couples to other fields like electrons and photons, and whose potential energy is minimised not at zero field value — the famous Mexican-hat potential — so the field rolls down to a non-zero minimum and sits there, affecting other fields. Anderson mentions this in words in one of his papers but didn't pursue it, partly because he was really a condensed matter physicist rather than a particle physicist, and partly, I think, because he recognised there would be a tremendous energy density associated with the field taking a non-zero value throughout empty space, and no one had ever noticed such a thing. He didn't connect it to gravity or the cosmological constant as far as I know, but it bothered him enough that he moved on to other things — and won the Nobel Prize for those instead.

Sean Carroll

By the 1960s, with Zeldovich, people had put the story together: the absolute value of the vacuum energy of a quantum field doesn't matter for particle physics — all that matters is how it changes from place to place — except when gravity comes into the game. Gravity cares about the total amount of energy in the universe, and Zeldovich appreciated that. He pointed out that we have a problem, because the natural scale for the vacuum energy is much bigger than it can possibly be, given the observations. An enormous discrepancy.

Sean Carroll

Let's jump to the modern way of thinking about this, which puts some meat on the bones of "natural values" — effective field theory. In this framework, you admit you don't understand everything: you don't know what's going on at very high energies, which in quantum field theory means very short distances. Happily, quantum field theory has the feature that even if you don't know what's going on at very short distances, you can still find an effective theory that only includes particles and phenomena at low energies and long distances — the infrared theory, as opposed to the short-distance ultraviolet. Following Ken Wilson and others, you say you have an ultraviolet cutoff, the shortest distance you claim to understand, and bundle up everything below that into what it does to your infrared theory. Some parameters in the infrared theory are relevant — they get more important at longer distances — and some are irrelevant. The cosmological constant, the vacuum energy, is super-duper relevant. It very much shows up in the infrared, and you can use effective field theory to ask how big we'd expect it to be.

Sean Carroll

The answer is hilariously bigger than it can be. I say this knowing that in the 1960s our cosmological observations weren't nearly as precise as today — they didn't need to be, because the discrepancy between the effective field theory prediction and what you actually observe is just so enormous that you don't need very precise observations to see it. The most dramatic way to say it: if you push your ultraviolet cutoff all the way up to the Planck scale, and compare the predicted vacuum energy to the observational limit — even the limit we had before we'd actually found the vacuum energy — the predicted value is bigger by the famous factor of 10 to the 122. One followed by 122 zeros. Not the number 122, which would be easy to deal with — one followed by 122 zeros. That discrepancy is known as the cosmological constant problem.

Sean Carroll

As I keep emphasising, you're a hundred per cent allowed to solve the cosmological constant problem by just saying: that's the way it is. The cosmological constant, in the infrared, is a measurable quantity — you had an expectation for what it should be, your expectation was wrong, tough cookies. That's a perfectly legitimate attitude, entirely compatible with the data — maybe that's just how things are, sometimes physics is like that. But a lot of theoretical physicists want to say maybe this huge discrepancy is a clue to some new physics we don't yet understand, rather than something you just measure and move on from.

Sean Carroll

By the 1980s, once the standard model of particle physics was in place, people turned their attention to gravity and cosmology — the 1980s were also when the first superstring revolution happened, and people really began thinking hard about the cosmological constant problem, having largely solved the problems in experimental particle physics. To be super clear: if the vacuum energy were anywhere close to its natural value from particle physics, the universe would have exploded apart into nothingness in a tiny fraction of a second after the Big Bang — the expansion rate would be so large you couldn't make an atom, let alone a molecule, planet, or star. It's not a delicate measurement with error bars you might be wrong about; the discrepancy is overwhelmingly obvious. And theoretically you don't even know the sign — it's a guess at magnitude, not really a prediction — and a large negative vacuum energy would make the universe recollapse in a tiny fraction of a second, also ruled out, since the universe has been around for roughly 14 billion years.

Sean Carroll

I entered graduate school in 1988, so for the rest of this podcast, and the next one, this is going to be a very Sean-centric view of these problems — not because I'm central to the story of dark energy, but because dark energy and the accelerating universe are central to me: I did a lot of work on it, wrote a lot of papers, and I'm not doing a lot of fresh research for these solo episodes, so I'm going to tell you what was happening around me at the time I was thinking about these things. There's a bunch of ideas from the late 1980s and early 1990s that people used to try to address the cosmological constant problem. I'm going to mention four attitudes or strategies. None of them are obvious slam dunks — that's why it's still an interesting problem.

Sean Carroll

One idea that got an enormous amount of attention is supersymmetry, which was incredibly popular in the 1980s especially, having grown out of the 1970s, connected to grand unification and string theory — the hot topic in theoretical particle physics. Supersymmetry, unlike other symmetries, does have something to say about the cosmological constant. For the rest of particle physics — the electroweak force, the strong force, the Higgs mechanism — you can ignore gravity and not worry about the cosmological constant, because only the changes in vacuum energy matter, not the absolute value. Supersymmetry is different because it's a mixture of space-time symmetry with internal symmetries — it relates particles of different spin, and spin is a space-time feature. So supersymmetry, in a rough sense, knows about space-time, which means it's related to both particle physics and gravity — exciting all by itself — and because it's related to space-time, it knows something about the vacuum energy too. In a supersymmetric theory, the vacuum energy is not arbitrary — it's determined by what's going on in the rest of the model, and people quickly realised there were versions of supersymmetry where the vacuum energy would be exactly zero.

Sean Carroll

They also realised you could get negative vacuum energy, which seemed weird, but they didn't worry about that too much at the time. What they did realise fairly quickly was a problem: supersymmetry isn't exact in the real world — if it were, there'd be a spin-zero partner of the electron with the same mass and charge, the "selectron," and no such particle exists. That's not a huge puzzle on its own; supersymmetry is presumably broken, and broken symmetries are easy to have in particle physics, that's what the Higgs mechanism does. You could even estimate the energy scale at which supersymmetry had to be broken. The problem is that once you break it, the vacuum energy generally comes out not just non-zero but incompatible with the data. So supersymmetry was intriguing for the cosmological constant problem, because it seemed to have implications for it, but the implications seemed to be bad — it took away your freedom to set the cosmological constant to whatever you wanted. That's not necessarily a sign you're on the wrong track — maybe you just hadn't filled in all the details — and people put a lot of work into finding a supersymmetric model that explained the small value of the cosmological constant, but the answer, so far, has been no.

Sean Carroll

Another avenue was wormholes and Euclidean quantum gravity — an idea that got very popular for a very brief period and then went away. The person who really put it together was Sidney Coleman, a professor at Harvard, my quantum field theory professor and a member of my thesis committee. A brilliant scientist, and very funny, and Sidney himself was a bit chagrined about this episode. His style was cautious — well-established calculations in quantum field theory, not speculation about crazy models — which is part of why he wasn't as famous as colleagues like Steven Weinberg or Sheldon Glashow. The one time he did speculate about a crazy model was this cosmological constant business, and it got him a lot of publicity, even though it ultimately went away. I want to tell you about it anyway, because even though it didn't work, it was really brilliant — if it had worked, it would have offered a super intellectually satisfying explanation.

Sean Carroll

People were beginning to think about gravity as well as particle physics, and one way in was the path integral approach to quantum gravity, developed by people like Stephen Hawking and Jim Hartle. The first step, already a suspicious one, is to treat space-time as four-dimensional space rather than space-time — Euclidean quantum gravity, all four dimensions treated alike, none picked out as time. In a path integral, you calculate a quantum mechanical probability by summing over all the ways you could get there — one of Feynman's ideas. So the proposal was to calculate the wave function of the universe by summing over all four-dimensional geometries with a boundary matching the universe we see. The simplest contribution is just a big four-dimensional sphere. Someone then realised: what if you have not one sphere but two, connected by a tiny wormhole with almost no contribution to the action, because it's tiny? Two spheres become almost as good a contribution as one — and then you can have three spheres, or two spheres with two wormholes, and so on, an infinite series to sum, exactly the kind of thing Coleman was really good at.

Sean Carroll

He and others — Steve Giddings, Andy Strominger, Joe Polchinski — worked on calculating the wave function of the universe as a sum over spheres connected by wormholes. People realised this was a real problem, because if a wormhole connects two spheres, it's not just gravity that gets involved — what if an electron falls down the wormhole and disappears from one universe to another? That would look like charge disappearing, which is bad. What Coleman figured out is that you can't just have a single electron go down the wormhole — you could have an electron and a positron go down together and a photon come out, conserving charge and energy. From far away, not knowing it's a wormhole, that looks just like an ordinary particle physics interaction. So the real effect of the wormholes, Coleman argued, is not to swallow particles, but to contribute to all the existing parameters of particle physics — the fine-structure constant, the Higgs mass, and the vacuum energy. And what he showed, by calculation, was that the wormhole contribution exactly cancelled the contribution of everything else. It was truly beautiful, and very convincing and provocative when it first appeared — this was the late 1980s, about when I arrived in grad school.

Sean Carroll

Sadly, the whole thing never lasted, because, as Coleman himself said very clearly — he wasn't bombastic about it — "this is a house doubly built on sand." We're talking about Euclidean quantum gravity and wormholes, things we don't understand very well at all, and the more people looked at it, the less sensible and reproducible it appeared. It didn't burn out so much as fade away — some people argued it actually makes the cosmological constant big rather than small, and it was very hard to pin down, because no one knows the rules. I'm just trying to give you a flavour of the kinds of ideas being bandied about at the time.

Sean Carroll

Another idea, which came along ten or fifteen years later, in the 2000s, is called self-tuning. I did write a little paper about this — I didn't invent the idea, other people did — once the notion arose that maybe we live on a brane: a lower-dimensional surface, say three-dimensional, embedded in higher-dimensional space, as you get in string theory. By carefully picking how the brane interacts with the rest of the world, you can find dynamics that cancel the vacuum energy on the brane. If you imagine a cosmological phase transition, ordinarily associated with a change in vacuum energy — the way the average energy of water and ice differs, which is why ice cubes float — self-tuning showed you could pick a set of fields such that a phase transition on the brane wouldn't change the effective, observed vacuum energy. It just goes away.

Sean Carroll

It turns out, again, that it didn't work — it's in the dustbin of history, a good idea that didn't quite pan out. I remember giving a talk on it at the University of Texas with Steven Weinberg in the audience. Weinberg was an expert on the cosmological constant — he'd written an extremely influential review article in the late 1980s surveying all the ways people had tried to solve the problem, and had proved a theorem saying you basically can't invent new fields to solve the cosmological constant problem without some fine-tuning creeping in, at least in ordinary quantum field theory. So when I explained self-tuning, he kept asking how it managed to get around that, and I eventually explained it, and he said: "That's not self-tuning, you tuned it." He was completely correct — you had to choose other parameters of the theory very carefully so that the vacuum energy would cancel.

Sean Carroll

My paper with Laura Mersini, written when I was a beginning professor at Chicago, wasn't about whether it was fine-tuned — I thought it was fine-tuned from the start — but about how the Friedmann equation, which normally feels the total energy density of matter and radiation, could be made to not feel the vacuum energy. What we showed was that the self-tuning mechanism effectively replaces the energy density rho in the Friedmann equation with rho plus p, the energy density plus the pressure. For vacuum energy, rho plus p is zero, because p equals minus rho. For ordinary matter, there's no pressure, so cosmologically you'd get an ordinary, matter-dominated universe at late times with the vacuum energy dropping out of the equation. The problem is radiation: in the early universe, before matter dominates, the universe is radiation-dominated, and radiation has positive pressure — a third of the energy density. Replacing rho with rho plus p turns that into something like four-thirds rho, which isn't allowed experimentally. I remember Shamit Kachru, a string theorist who was one of the people who first proposed self-tuning, sitting in the audience when I gave a talk at Stanford, saying, "You're telling me I've made a prediction that could be experimentally ruled out? That's awesome." It was ruled out — and it wasn't theoretically that attractive either, in the end.

Sean Carroll

I'll give you one last attempt at solving the cosmological constant problem — you know this one, the anthropic principle. It was suggested by a number of people, but Steven Weinberg was the one who did it carefully. He showed that if you live in a multiverse, with different parts having different values of the cosmological constant but everything else the same, you would naturally not find a large cosmological constant in the parts hospitable to life — because if it's too big, it blows galaxies apart before they can form, and if it's negative and too big, it collapses the universe too quickly. There's a small window. To his eternal credit, Weinberg made this prediction in the 1980s, before the cosmological constant was discovered, saying that if the anthropic principle is on the right track, we should expect a non-zero value, because there's no symmetry forcing it to be exactly zero. It turned out to be right, and he was more or less in the right ballpark. That doesn't mean the anthropic principle is correct, but it does mean that, compared to the other ideas I've described, it did a better job of making a prediction and getting it right.

Sean Carroll

So, circa 1990, we had the idea of the cosmological constant, we'd related it to the vacuum energy, and we'd tried and largely failed to explain why it should be small. That's where I come into the story. Bill Press, a professor at Harvard, then at the University of Texas — a well-respected theoretical cosmologist in the astronomy department, and one of the authors of Numerical Recipes, which was, before you could ask an LLM to write your code for you, a very helpful book for scientific computation — had been invited by Allan Sandage, editor of Annual Reviews of Astronomy and Astrophysics, to write a review article on the cosmological constant: the astronomy-flavoured version, since Weinberg had already done the particle physics one. Bill asked me, a grad student in the astronomy department at the time, to collaborate, mainly because he didn't want to write about supersymmetry or wormholes himself.

Sean Carroll

Bill did a wonderful job figuring out the effect of the cosmological constant on cosmological observations — growth of large-scale structure, and a million other things — deriving formulas and semi-analytical approximations that were genuinely useful to working cosmologists. Neither of us was an expert on the observational side, so we asked Ed Turner, an astronomer at Princeton, to help. Here's where it gets a little embarrassing: Ed's real expertise was statistics of strong gravitational lensing — the probability that a far-away galaxy or quasar gets lensed by an intermediate galaxy, which changes as a function of the cosmological constant. When we started writing around 1990, this was thought to be the best way to constrain it. As it turned out, within ten or fifteen years, two quite different methods proved decisive — supernova distances, and anisotropies in the cosmic microwave background — and we mentioned neither in our review. Right place, wrong time. We were superseded, but useful in the meantime.

Sean Carroll

My job was to write about the theory side — the anthropic principle, wormholes, and so on, but also the naturalness and fine-tuning issues. There's the cosmological constant problem, why it's so small, but there's also a separate question: what if it's not exactly zero? In 1990, we didn't know it wasn't zero, but we were open to the possibility, even though most theorists, myself included, thought it probably was zero. The prevailing intuition was that, among all the theories we hadn't yet thought of, more of them naturally give exactly zero than give ten-to-the-minus-122 times the natural value but not ten-to-the-minus-123 — maybe there's a symmetry or mechanism that forces it to zero. The one big counter-example was the anthropic principle.

Sean Carroll

There was one thing everyone recognised: given the Hubble constant, there's a certain energy density compatible with a spatially flat universe — the critical density, the special, middle point between a positively curved sphere and a negatively curved saddle. In the early 1980s, the inflationary universe scenario, from Alan Guth and others, explained why the universe is so smooth, homogeneous, and isotropic by proposing a period of superfast early expansion. In 1990, inflation's main prediction beyond that was that the density should be the critical density — spatially flat. But astronomers, measuring the gravitational density of galaxies and clusters, kept getting numbers around 0.3 of critical density, so many thought the universe was negatively curved rather than flat. One motivation for taking the cosmological constant seriously was that the universe might still have the critical density overall, with roughly 0.3 of it matter and roughly 0.7 the cosmological constant — allowed by the data at the time, and a good story for inflation, but weird for another reason: the coincidence problem.

Sean Carroll

The universe has been expanding for a long time — 14 billion years — and as it expands, matter and radiation dilute away: the density of matter and radiation per unit volume goes down as the volume goes up, because the number of galaxies stays the same. But the vacuum energy density stays exactly fixed as the universe expands. So if vacuum energy and matter density are roughly comparable today, then in the past there was vastly more matter than vacuum energy, and in the future there'll be vastly more vacuum energy than matter — we just happen, coincidentally, to live at the one moment in cosmic history when the two are comparable. That's the coincidence problem, and I was quite taken with it — I made plots for the review article showing how hilariously unlikely it would be for the cosmological constant to be the same order of magnitude as the matter density purely by chance. But no amount of theoretical discomfort settles the question — eventually you have to go look at the data.

Sean Carroll

In 1992, the COBE satellite discovered anisotropies in the cosmic microwave background. The background radiation itself had been discovered in the 1960s, and between then and the 1990s it looked perfectly smooth in every direction. We knew it couldn't be perfectly smooth, because the galaxies in today's universe had to come from somewhere — tiny ripples at early times — and eventually we'd find them, and COBE did. I remember it vividly — one of the first times in my life I felt like an insider, because Bill Press, an inveterate gossip as well as a genuine insider, knew before the official announcement and told everyone at lunch, and I got to pass the news on to people like Sidney Coleman in the physics department. Jim Peebles and Zeldovich and others had worked out the theory of what those anisotropies should look like back in the 1960s and '70s, but the rest of the community hadn't really absorbed what could be done with the data once it came in. COBE itself was primitive — it found that the anisotropies were there, without resolving much detail — but it was clear that future telescopes would bring enormously richer data, and it turned out those data would be very informative about cosmological parameters: the Hubble constant, the cosmological constant, the dark matter density.

Sean Carroll

The other decisive strand of data was the Hubble diagram — distance versus redshift for distant objects — measured in a new way, using Type Ia supernovae. I had nothing to do with that work myself, except that I was plugged in; my friends were deeply involved. There are different classes of supernovae — twos, ones, with subdivisions of the ones. My office mate, Brian Schmidt, a graduate student of Robert Kirshner at the Harvard Astronomy Department, worked on Type II supernovae and the Hubble constant, using the expanding-photosphere method: modelling the expanding supernova with geometry and blackbody radiation to compare brightness and distance. It was decent but not quite good enough, given the uncertainties in the sphericity of a Type II event.

Sean Carroll

The alternative was Type Ia supernovae, whose physics was, and to some extent still is, less well understood, but which seemed to be almost standard candles — nearly the same intrinsic brightness. There's a reason for that: a Type II supernova happens when a red giant runs out of fuel, collapses, and bounces, and different red giants differ. A Type Ia supernova happens when a white dwarf, fed by a companion star gradually dribbling mass onto it, hits the Chandrasekhar limit, collapses, and explodes — and the Chandrasekhar limit is universal, the same for every white dwarf, which is why Type Ia supernovae should all be roughly the same brightness. And they're very bright, which helps. They're not exactly the same brightness, which matters, but approximately — enough to gauge relative distance.

Sean Carroll

Bob Kirshner also worked on Type Ia supernovae, and the graduate student he put on that task was Adam Riess, on the floor below us. Harvard, where I was a grad student, was effectively the centre for cosmological supernova observations at the time, but the initial focus was measuring the Hubble constant, not the cosmological constant. There was an idea, though: what if, instead of waiting around for supernovae to happen by accident, you ran a dedicated search programme? That idea, tellingly, didn't come from the astronomers — it came from the particle physicists. Saul Perlmutter, in particular, asked why you'd just wait for supernovae rather than getting a big telescope, surveying a lot of galaxies, and finding a systematic, predictable supply of them. Particle physicists think big; they wanted to be systematic across the whole sky. I had long phone conversations with Saul and his co-author Ariel Goobar about whether this could measure the cosmological constant, because I was still a grad student, but I'd been an author on the review article with Bill Press and Ed Turner, which they knew about — and I was friends with Bob, Brian, and Adam as well.

Sean Carroll

Saul and his collaborators formed the Supernova Cosmology Project — largely ex-particle physicists who had the ability to secure Department of Energy funding and strong computing expertise, and who just went out and did it. The astronomers, Bob Kirshner very much included, were sceptical it would ever work — for good reasons, not just protecting their turf. They knew the pitfalls: not all Type Ia supernovae are quite the same, and they had data showing it. Bob and others, with Adam involved, and I think Bill Press too, had pioneered a method to standardise the candles: not all Type Ia supernovae have exactly the same brightness, but the time it takes them to brighten and dim is related to their absolute brightness — the Phillips relation, after Mark Phillips. So they were trying to reduce a heterogeneous sample down to a standardisable one. And they were impressed enough with the difficulties that they doubted this Perlmutter thing would ever work.

Sean Carroll

My understanding here is incomplete, because it's just from my own vantage point, but it was Nick Suntzeff and Brian Schmidt — Brian and Adam had by then both graduated and moved on — who said, actually, we think we can do this too, and catch up, because we know astronomy better than the particle physicists do. Saul's group started first but needed to learn the astronomy; the group that became known as the High-Z Supernova Team — Z for redshift — started later but caught up, reducing data very effectively, and Adam and Bob and others joined it as well. They were both right: Saul was right that it could be done and got it started, and Brian, Adam, Bob, and Nick were right that you needed to be careful and get it correct.

Sean Carroll

I have a cute memory of getting a set of papers in the mail from the astronomy group — the High-Z Supernova Team — advertising their work, bound in a spiral notebook, titled: "Measuring the deceleration of the universe." The irony being that what they actually ended up measuring was the acceleration, which no one expected at the time — they thought they'd measure the Hubble constant, and maybe the density parameter, telling us whether we were at the critical density or not.

Sean Carroll

In 1997, I was a postdoc at the Institute for Theoretical Physics in Santa Barbara, working on various things with topological defects and extra dimensions. Phil Lubin, an observational cosmologist there, organised a workshop in December 1997 on determining cosmological parameters — Hubble constant, cosmological constant, and so on — from the cosmic microwave background. They didn't yet have the data to do it, but they were working on how. Phil asked me to give a talk on measuring cosmological parameters by every method that wasn't the microwave background — galaxies, supernovae, that sort of thing. I said sure, and being forced to prepare it, I went through a huge amount of the modern observational literature, which was informative: over and over, from different angles, the data were telling us that the theorists' favourite model — a flat universe made entirely of ordinary matter at the critical density, exactly what inflation predicted — couldn't be right. I gave the talk and said: I don't know what's wrong, but there's already enough data in 1997 to rule that model out. Maybe the universe is open, without enough matter for critical density. Maybe there's a cosmological constant. Maybe dark matter is warm or mixed rather than cold. Maybe there's a tilt in the spectrum. I didn't know which, but something was going on. That was the mood in cosmology at the time.

Sean Carroll

In February 1998, the High-Z Supernova Team, led by a paper with Adam Riess as first author, reported a non-zero cosmological constant. A couple of months later, Perlmutter's group announced they were getting exactly the same answer. This was a dramatic claim — Adam and others have said in interviews that they didn't believe it themselves at first; it's easy to make a mistake, and you don't want to publish "the universe is accelerating" unless you're sure you're on the right track. But they couldn't make it go away, so they published. I had a bit of an inside scoop, being friends with everyone involved, and got to give talks in Santa Barbara spreading the news, and everyone was hugely excited. People were willing to accept it quickly precisely because it made everything else snap into place — which is an interesting lesson in the history of science. A dramatic claim that flies in the face of everything we think we know is correctly met with scepticism. A dramatic claim that makes everything else fit together is much more readily believed. Not everyone accepted it right away, and that's fine — there's always room in physics for a few grumpy holdouts, as long as they eventually accept it once the data become overwhelming.

Sean Carroll

In the early 2000s, cosmic microwave background data, from ground-based telescopes and the WMAP satellite, confirmed the picture the supernova data had put together: 0.7 of the critical density as vacuum energy, the cosmological constant, and 0.3 as matter — just as the astronomers had been saying all along. That's still the best fit, twenty years later — lambda-CDM cosmology, lambda for the cosmological constant, CDM for cold dark matter. It fits very well, though not perfectly — we've discussed with both Adam Riess and Marc Kamionkowski, elsewhere on Mindscape, things like the Hubble tension that don't completely fit in. That's what drives progress in science.

Sean Carroll

So where does that leave us? The data fit well, but the coincidence problem — that we live in the unique era when matter and vacuum energy are both relevant — is still very much with us. I think about it this way: the best solution we have right now is the anthropic principle, and it's not a good solution — it's messy, involves a lot of speculative physics, and there's plenty we don't understand theoretically about it. It's still better than any of the alternatives. Not that it's right, but that's the state of the art.

Sean Carroll

The cosmological constant problem, separately from the coincidence problem — why the vacuum energy is so small — has only got harder now that we know it's not exactly zero. It was easier, in most theorists' minds, to imagine a mechanism that sets it all the way to zero than one that sets it to ten-to-the-minus-122 times the Planck scale, but that's apparently where it is. It might be a hint that the whole effective field theory paradigm fails once gravity and cosmology enter the picture — which doesn't sound so crazy said casually, but is genuinely puzzling, because even gravity is supposed to fit into the effective field theory framework from everything else we know. Some people have boldly pushed further: Tom Banks and Willy Fischler, whom I've mentioned before, argue that the dimensionality of Hilbert space is a finite number, and that this changes the cosmological constant problem entirely, because the dimensionality of Hilbert space is related to the cosmological constant and so isn't an ordinary parameter in an effective field theory. I'm saying these words without expecting you to fully follow them, just to let you in on how some physicists are thinking about the big issues right now. It's only one number we've measured, so it's hard to build detailed models on top of it — but that one number is spurring new ideas.

Sean Carroll

If the cosmological constant is truly constant, that tells us something about the future of the universe. When I was younger, we still took seriously the idea that the universe might recollapse someday. If it's truly a cosmological constant making the universe accelerate right now, there's no reason for that to stop — the density of energy in empty space is a number, roughly 10 to the minus eight ergs per cubic centimetre, and it will never go away. The universe will get emptier, colder, and more desolate, forever. So enjoy the time you have. That would be the lesson.

Sean Carroll

Or, the other possibility: maybe the cosmological constant is zero after all, and we were right back in the 1980s that, among the ideas we haven't yet come up with, it's easier to find one that sets it to zero than to some small but non-zero number. You might object that the data don't allow that — but that's not quite true. The data say there's something making the universe accelerate; they don't insist that thing is the cosmological constant specifically. The cosmological constant is a very good fit to the data right now, but we were surprised when we found it, so we should stay open-minded, and consider that what's making the universe accelerate might be something else — dynamical dark energy, which can change with time just a little, unlike the strictly constant cosmological constant. That opens up an enormous number of possibilities, which is where the next episode picks up.