Sean Carroll on Dark Energy, the Cosmological Constant, and the Accelerating Universe

Author: Sean Carroll

Physicist Sean Carroll works through the theories built to explain why the universe’s expansion is speeding up, and what the mysterious ‘dark energy’ behind it might actually be. This is Part 2 of a two-part solo episode — Part 1, Sean Carroll on Vacuum Energy, the Cosmological Constant Problem, and the Anthropic Principle, covered vacuum energy and the cosmological constant, the background needed for this instalment, though Carroll designed each part to stand on its own.

Key ideas

1. The universe is accelerating, and a cosmological constant explains why. Supernova surveys in the late 1990s found that distant galaxies are receding faster than gravity alone would allow — the expansion of the universe is speeding up, not slowing down. A cosmological constant, Einstein’s old vacuum-energy term, accounts for this because vacuum energy carries negative pressure equal in size to its own energy density. Fed into general relativity, negative pressure acts gravitationally repulsive rather than attractive, stretching space apart at an ever-increasing rate.

2. Two puzzles remain unsolved: the cosmological constant problem and the coincidence problem. Effective field theory — the framework physicists use to combine quantum field theory with particle physics — predicts a vacuum energy density far larger than what telescopes actually measure; the observed value is smaller than the theoretical expectation by something like 10 to the 122. Separately, the coincidence problem asks why the vacuum energy and the matter density happen to be within a factor of a few of each other today, when the matter density dilutes towards zero as the universe expands while the vacuum energy stays constant — so their ratio has differed wildly at every other point in cosmic history.

3. Dynamical dark energy (quintessence) was proposed on almost no evidence. Rather than a strict, unchanging cosmological constant, some physicists proposed a slowly rolling scalar field — pictured as a ball inching down a very shallow hill — whose potential energy mimics a constant while its kinetic energy stays negligible. Carroll is blunt that when this idea (dubbed quintessence by Caldwell, Dave and Steinhardt around 1998) first appeared, ‘there was essentially zero empirical reason to do so’; the plain cosmological constant already fit the data perfectly, and the motivation was theoretical open-mindedness after being surprised once by the discovery of acceleration itself.

4. Whatever dark energy is, it must be almost smooth in space and almost constant in time. If dark energy clumped into galaxies and clusters the way matter does, it would show up in the cosmic microwave background, gravitational lensing and galactic orbits — exactly the signals that were absent before the supernova discoveries. Cosmologists track how close to constant a candidate is using the equation-of-state parameter, w, the ratio of a substance’s pressure to its energy density: w = −1 exactly for a true cosmological constant, and closer to −1 means slower fading. A workable quintessence field also turns out to need an extraordinarily tiny mass — roughly the Hubble constant expressed in energy units, far below the mass of any known particle — an extreme fine-tuning that echoes the naturalness worries behind particle physics’ own hierarchy problem.

5. Modifying gravity itself is the alternative to inventing new dark stuff. Instead of adding an unexplained energy field, some theories change Einstein’s equations directly — Carroll’s own attempt added a 1/R term to the gravitational action, work that fed into what became known as f(R) gravity. This sits alongside the standard cosmic budget: roughly 70% of the universe’s critical density is dark energy and about 30% is matter (25% dark matter, 5% ordinary). Dark energy and dark matter share a label but nothing else — dark matter clumps and behaves like ordinary matter gravitationally; dark energy does not clump and pushes space apart.

6. Decades on, the honest answer is that nobody knows. The plain cosmological constant still fits every observation, and Carroll says ‘the smart money’ remains on it. Recent hints from the DESI survey suggest the dark energy’s equation of state might not be exactly −1 and could even be evolving with time, but Carroll is sceptical the current data justifies that conclusion — the fits, he thinks, are too quick. He treats the question as genuinely open, calling the discovery that the universe accelerates ‘the single most surprising and profound discovery in fundamental physics’ of his career.

Context

Carroll narrates much of this as a participant, not just an explainer: his own contribution to the 1999 Garnovich et al. paper on constraining w, his work with Mark Trodden and Mark Hoffman ruling out ‘phantom energy’ (w < −1) as catastrophically unstable, and his early, ultimately-shelved 1/R modified-gravity idea that others independently rediscovered. The running thread is candour about how little of this theoretical effort has actually paid off — quintessence didn’t solve the coincidence problem, phantom energy proved unstable, and f(R) gravity does not explain the flat galactic rotation curves usually attributed to dark matter.

See also

• Sean Carroll — host
• Lisa Randall on Dark Matter, Physics, and Extinction — the other ‘dark’ component of the cosmic budget, and why it is a wholly different phenomenon from dark energy
• Don Lincoln on Particle Physics, Antimatter, and the Biggest Mysteries in Physics — the effective field theory and particle-physics naturalness reasoning behind the cosmological constant problem
• Daniel Harlow on Quantum Gravity, Black Hole Information, and the Holographic Principle — quantum gravity approaches to the puzzles vacuum energy raises
• Janna Levin on Black Holes, Wormholes, and Quantum Mysteries — general relativity’s other great open frontiers