It outweighs everything else in existence, it governs the fate of the Universe and it cannot be explained by known physics.

Dark energy is the name physicists use for whatever substance, force or property of space is messing with the Universe, making its expansion accelerate. As yet, we know almost nothing about it, which has allowed theories about it to multiply uncontrolled. But astronomers are training an impressive array of instruments on the problem. Telescopes and radio detectors are tracing out more and more of the Universe in the hope of finding a fingerprint of dark energy. Space agencies are planning two missions to examine the stuff. The most violent of all stellar explosions could provide insight into its influence on the early Universe. And a new branch of astronomy may also have a part to play, as gravitational-wave detectors begin to listen for the effect of dark energy on the echoes of colliding black holes.

 

Unlike the search for dark matter, this quest is a young one. Scientists have known since the late 1920s that the Universe is expanding, but it was assumed that expansion must be slowing down, as the force of gravity between galaxies and other matter puts on the brakes. In 1998, two teams found quite the opposite. They had been searching for stellar explosions of a particular type—type Ia supernovae, which occur when white-dwarf stars undergo a runaway nuclear reaction. The intrinsic brightness of a type Ia is fixed by how fast its light fades—brighter ones burn more briefly. So by counting how many days a type Ia takes to fade, you can work out how much light the explosion emitted; then, by measuring its apparent brightness at Earth, you can calculate how far away the supernova really is and how long the light has been travelling. This type of cosmological probe is called a standard candle.

Astronomers also measure each supernova's redshift. This is the amount by which the wavelength of light has been stretched out since it was emitted, which reveals the degree by which space has expanded. Combining these observations allows astronomers to determine the expansion of the Universe over time—that is how both teams discovered that the speed of expansion is not slowing, but accelerating. Their conclusion: something seems to be overpowering the attraction of gravity.

An enduring puzzle

That something, now known as dark energy, is shrouded in mystery. All we know is that it has the peculiar property of pushing outwards, unlike gravity, which was thought to be the dominant cosmic force. Now astronomers want to find out whether this enigmatic phenomenon changes across time. They have begun to look even more closely at how the Universe has expanded—some by refining the use of supernova standard candles, others by devising new cosmological tools.

So far, the most effective tool is based on cosmic sound waves. Shortly after the Big Bang, the Universe was filled with an elastic mixture of ions, electrons and radiation. Small density anomalies (created by quantum fluctuations in the first 10−32 seconds of the life of the Universe) gave this cosmic bell a tap, sending sound waves rippling outwards. After about 400,000 years, the Universe had cooled enough for ions to capture loose electrons. Because the resulting neutral atoms were transparent to radiation, letting photons whizz by them, the mixture was no longer elastic. And because sound needs an elastic medium to travel, the primordial sound waves were halted, imprinting an indelible pattern on the large-scale structure of the Universe. So instead of being positioned entirely at random, galaxies have a slight tendency to be spaced at regular intervals. The characteristic distance has been growing as the Universe expands, and stands at about 500 million light years (153 megaparsec) today.

 
Click to enlarge. Credit: Nature, September 28, 2016, doi:10.1038/537S201a

Just as supernovae work as standard candles, these baryon acoustic oscillations (BAOs) can act as standard rulers. Mark the position of enough galaxies and you can measure the apparent size of BAOs. Compare that with the size predicted by their redshift, and you can work out how far away these particular BAOs are. By measuring the redshift of these galaxies, and plotting that against distance, it is possible to reveal how the expansion of space has behaved through cosmic history.

The best view yet of BAOs was revealed in July by a Sloan Digital Sky Survey programme called the Baryon Oscillation Spectroscopic Survey (BOSS). This is the largest such galaxy survey yet. “This technique is really coming into its own,” says Saul Perlmutter, a physicist at the University of California, Berkeley, who led one of the teams that discovered dark energy in 1998 and who received a share of the 2011 Nobel Prize in Physics along with Adam Riess and Brian Schmidt for the work.

As well as backing up the supernova results with independent evidence that expansion is accelerating, the BOSS data give some clues about how dark energy behaves. And the pattern of acceleration suggests that if dark energy is changing, it is not changing very fast.

For now, that's a conclusion that seems to favour a candidate for dark energy known as the cosmological constant. In the 1920s, Einstein toyed with adding a constant term to his equations of general relativity—equivalent to giving empty space its own energy. According to general relativity, this cosmological constant would indeed oppose the force of ordinary gravity. Einstein originally tuned the value of the constant to create a balanced, static model Universe. But in 1929, Edwin Hubble showed that distant galaxies are receding from us, and astronomers realized that the Universe is actually expanding. Einstein ditched the constant. Now, however, with evidence that Universal expansion is accelerating, the cosmological constant has come back into contention.

Energetic mystery

The question is why the vacuum of space should have energy at all. Quantum-field theory posits a profusion of virtual particles that briefly come into existence and then disappear—a seemingly outrageous idea, but one that has allowed quantum theorists to make extremely accurate predictions of how ordinary particles interact. These virtual particles could be behind dark energy's repulsive force.

But it's hard to make the numbers stack up. The vacuum energy needed to produce the observed cosmic acceleration is about 1 joule per cubic kilometre of space; the simplest version of quantum-field theory adds up the energy of those virtual particles to give a value about 120 orders of magnitude higher than that. Such dense vacuum energy would rapidly rip the Universe to shreds, and plainly that has not happened.

Perhaps scientists are missing something. As-yet-undiscovered particles could cancel out the energy supplied by known particles. But, although it is simple to devise a theory that makes the value zero, it is hard to almost-but-not-exactly cancel out a huge number to leave the small required value of vacuum energy. “The cosmological constant is an odd beast,” says Perlmutter. “It makes the theory seem bizarrely asymmetric.”

So, although the cosmological constant remains the front-runner, theorists have been busy devising alternative forms of dark energy. Some have created new theories of gravity, similar to general relativity, but generating repulsion on very large scales. Others posit some kind of space-filling fluid, sometimes called quintessence, which acts a little like the cosmological constant, but slowly changes in density. Whatever the answer, dark energy is key to opening a window on “a completely unexplored region of fundamental physics,” says Mark Trodden, a theoretical cosmologist and director of the Penn Center for Particle Cosmology in Pennsylvania, Philadelphia. Finding the answer would not only change the view of nature, but also foretell the fate of the Universe.

Star surveys

A slew of projects are preparing to gaze more deeply into the dark enigma, and work out whether dark energy really has always been the same across the Universe. The Dark Energy Survey (DES) has already begun, using the Victor M. Blanco telescope in Chile to scan a swathe of the southern sky, observing supernovae and cataloguing more than 200 million galaxies. Early in 2017, an even bigger survey—the Javalambre Physics of the Accelerating Universe Astrophysical Survey (J-PAS) near Teruel, Spain—should start drawing its own 3D map of the Universe to reveal BAOs. It will cover much of the northern sky and analyse up to 500 million galaxies with an innovative instrument that uses 56 colour filters to reveal redshift.

Meanwhile, in western Canada, a very different instrument is beginning to take shape. The Canadian Hydrogen Intensity Mapping Experiment (CHIME) near Penticton in British Columbia is an unusual radio telescope built from a series of half pipes, like a giant skateboard park. It gathers radio waves from along a north–south line that sweeps around as Earth rotates to build up a picture of the sky.

CHIME is built to pick up the waves emitted by cool hydrogen gas. Like galaxies, this carries the imprint of ancient acoustic oscillations. It might even reveal BAOs better than galaxy surveys, because galaxies are the result of relatively complex processes, whereas gas follows the original sound waves more directly. “It is a very clean measurement,” says principal investigator Mark Halpern. And its simplicity makes the instrument relatively cheap at Can$10 million (US$7.8 million). “CHIME is a stunning bargain,” says Halpern, an experimental cosmologist at the University of British Columbia in Vancouver, Canada.

Over the next decade, spacecraft and giant ground-based telescopes with much bigger budgets are expected to join the hunt for dark energy. Between them, these big projects, along with DES and J-PAS, will wield four cosmic tools. As well as spotting supernovae and plotting BAOs, they will also measure gravitational lensing and catalogue clusters of galaxies. Clusters are pulled together by gravity, so their growth could reveal whether the force of gravity begins to change at large scales. And gravitational lensing, the bending of distant images by intervening matter, produces subtle patterns in the orientation of galaxies. Seeing how these patterns vary over cosmic time could reveal changes in dark energy.

Starting in about 2023, the Large Synoptic Survey Telescope (LSST) in Chile promises to find huge numbers of supernovae and pinpoint billions of galaxies to trace BAOs, cluster growth and lensing. “LSST will be like DES on steroids,” says DES director Josh Frieman. The European Space Agency's Euclid mission, scheduled to launch in 2020, will also make use of gravitational lensing. Above the blurring caused by Earth's atmosphere, Euclid's sharp vision will be better able to spot the orientation of lensed galaxies. It will also be able to collect near-infrared light that is blocked by the atmosphere. NASA plans to launch a similar mission, WFIRST, sometime in the mid-2020s. WFIRST will have even sharper vision than Euclid, because it is based around a larger, 2.4-metre mirror—a piece of optics donated by the US National Reconnaissance Office, which runs the country's intelligence satellites. “That was an unusual technology-repurposing game,” says Perlmutter.

Burst of ideas

Even with so many eyes on the sky, dark energy may remain elusive. So some astronomers are looking at more outlandish probes of the cosmos. Gamma-ray bursts (GRBs) are flashes of high-energy radiation from the distant Universe. Many are thought to be caused when the core of a massive star collapses to form a black hole or neutron star. At Stanford University in California, Maria Dainotti wants to use GRBs as a new type of standard candle. This seems like a difficult task because these bursts are infamously diverse, flashing and fading seemingly without any pattern. “If you have seen one GRB, you have seen one GRB,” says Dainotti. But in 2008, she found that among certain GRBs, for which emission falls to a plateau and then drops off again, a shorter plateau means a brighter burst (M. G. Dainotti et alMon. Not. R. Astron. Soc. 391,L79–L83; 2008).

Dainotti is cautious about using GRBs for precision cosmology just yet, partly because there is not yet a clear physical reason for her correlations. Researchers don't yet know what is going on inside GRBs when the star's core collapses—the high-energy emission could be generated by a rapidly spinning neutron star or by material falling into a newborn black hole.

But when the theory is better established, bursts such as this could illuminate the early days of dark energy. GRBs are much brighter than type Ia supernovae, so they could be used to see farther and trace expansion back to when the Universe was less than one billion years old. If dark energy is changing its nature, then that distant view may be crucial.

Gravity drops in

It may be that a new kind of astronomy will be needed to crack the dark enigma. In 2016, the Laser Interferometer Gravitational-Wave Observatory (LIGO) collaboration finally announced the detection of travelling distortions of space-time known as gravitational waves, predicted by Einstein a century ago. A distinctive chirp lasting a fraction of a second was the echo of colliding black holes, which more than one billion years ago shook the fabric of space-time as the black holes spiralled in and merged with each other.

“With gravitational waves like these, we could measure distance,” says Stephen Fairhurst, a physicist at the University of Cardiff, UK, and a member of the LIGO collaboration. The shape of the waves reveals the mass of the black holes and the total energy emitted. Combine that with the strength of the waves as they reach Earth, and you can work out the distance.

However, plotting the expansion history would also require finding the redshift, which is trickier. It may be possible to find the host galaxy of one of these events and use its light to reveal the redshift, although the host could be one of many galaxies in a broad target area, because gravitational-wave detectors cannot yet pinpoint direction precisely. If all these many and varied tools find no change in the behaviour of dark energy, researchers may have little choice but to give up and embrace the cosmological constant. “The moment we find ourselves accepting the cosmological constant will be when theory makes a convincing predictive step,” says Perlmutter. For example, a theory might predict a new class of particles to curb the cosmological constant, and these particles might then be detected by the Large Hadron Collider at CERN, Europe's particle-physics laboratory.

Physicists are fond of weirdness, so most will probably be hoping for the other outcome: that most of the substance of our Universe is an evolving thing that is even stranger than vacuum energy. If the source of acceleration is found to be either a new energy field or a modification of gravity, the consequences will be profound. “It could cause us to rethink how gravity and particle physics interact,” says Trodden. Finding a particle-based description of gravity has obsessed theoretical physicists since Einstein. To finally manage it, we may have to abandon his cosmological constant for the second time.

This article was originally published as part of Nature Outlook: The dark universe, a supplement to Nature. The Outlook was sponsored by Eisai Inc. All Nature Outlook content is editorially independent of sponsors, unless explicitly labelled as promotional content.