On a clear day in Gibraltar, looking out of Gorham's Cave, you can see the rugged northern coast of Morocco looming purple above the turquoise sea. Inside the cave, quiet prevails, save for the lapping of waves against its rocky beach. But offshore, the strait separating the Iberian Peninsula from the African continent bustles with activity.* Fishing vessels troll the waters for tuna and marlin, cruise ships carry tourists gawking at Gibraltar's hulking limestone massif, and tankers ferry crude oil from the Mediterranean to points west. With its swift, nutrient-rich currents, mild climate and gateway location, the area has attracted humans for millennia.

One impressive group dwelled in the region for tens of thousands of years, weathering several ice ages here. During such times lower sea levels exposed a vast coastal plain in front of the cave, land that supported a variety of animals and plants. These individuals cleverly exploited the local bounty. They hunted large animals such as ibex and seals and small ones such as rabbits and pigeons; they fished for bream and gathered mussels and limpets from the distant shore; they harvested pine nuts from the surrounding evergreens. Sometimes they took ravens and eagles for their plumage to bedeck themselves with the beautiful black flight feathers. And they engraved their cave floor with symbols whose meaning has since been lost to time.

In all these ways, these people behaved just like our own Homo sapiens ancestors, who arose in Africa with the same anatomy we have today and later colonized every corner of the globe. But they were not these anatomically modern humans. They were Neandertals, our stocky, heavy-browed cousins, known to have lived in Eurasia between 350,000 and 39,000 years ago—those same Neandertals whose name has come to be synonymous in pop culture with idiocy and brutishness.

The scientific basis for that popular pejorative view has deep roots. Back in the early 1900s the discovery of the first largely complete Neandertal skeleton, from the site of La Chapelle-aux-Saints in France, gave rise to the group's image problem: deformities now known to reflect the old age of the individual were seen as signs of degeneracy and subhumanness.

Since then, the pendulum of paleoanthropological opinion has swung repeatedly between researchers who see Neandertals as cognitively inferior to H. sapiens and those who see them as our mental equals. Now a rash of new discoveries is fanning the debate. Some fossil and ancient DNA analyses seem to suggest that Neandertal brains were indeed different—and less capable—than those of H. sapiens. Yet mounting archaeological evidence indicates that Neandertals behaved in many of the same ways that their anatomically modern contemporaries did.

As scientists advance into the Neandertal mind, the mystery of why our closest relatives went extinct after reigning for hundreds of thousands of years is deepening. The race is on to solve this extinction riddle: such insight will help reveal what it was that distinguished our kind from the rest of the human family—and set anatomically modern humans on the path to becoming the enormously successful species we are today.

Bony Inklings

Paleoanthropologists have long sought clues to Neandertal cognition in the fossilized skulls they left behind. By studying casts of the interior of the braincase, researchers can reconstruct the external form of an extinct human's brain, which reveals the overall size as well as the shape of certain of its regions. But those analyses have failed to turn up much in the way of clear-cut differences between Neandertal brains and those of H. sapiens. (Some experts think Neandertals were just another population of H. sapiens. This article treats the two groups as different human species, albeit very closely related ones.) Neandertal brains were a little flatter than ours, but they were just as big—indeed, in many cases they were larger, explains paleoneurologist Ralph Holloway of Columbia University. And their frontal lobes—which govern problem solving, among other tasks—were almost identical to those of H. sapiens, judging from the impression they left on the inside of the braincase. That impression does not reveal the internal extent or structure of those key brain regions, however. “Endocasts are the most direct evidence of brain evolution, but they are extremely limited in terms of giving you solid information about behavior,” Holloway admits.

In a widely publicized study published in 2013, Eiluned Pearce of the University of Oxford and her colleagues purportedly got around some of the limitations of endocasts and provided a way of estimating the size of internal brain areas. The team used eye-socket size as a proxy for the size of the visual cortex, which is the brain region that processes visual signals. They found that the Neandertal skulls they measured had significantly larger eye sockets than modern humans have—the better for coping with the lower light levels available in their high-latitude homes, according to one theory—and thus larger visual cortices. With more real estate dedicated to processing visual information, Neandertals would have had less neural tissue left over for other brain regions, including the ones that help us maintain extensive social networks, which can buffer against hard times, the researchers argued.

Holloway is not convinced. His own endocast work indicates that there is no way to delineate and measure the visual cortex. And Neandertal faces are larger than those of anatomically modern humans, which might explain their larger eye sockets. Moreover, people today are hugely variable in the proportion of visual cortex they have relative to other brain regions, he observes, and this anatomical variability does not appear to correspond to differences in behavior.

Other fossil analyses have yielded similarly equivocal signals about the Neandertal mind. Studies of limb asymmetry and wear marks on tools as well as on the teeth (from using them to grasp items such as animal hides during processing) indicate that Neandertals were as right-handed as we moderns are. A strong tendency toward favoring the right hand is one of the traits that distinguishes H. sapiens from chimpanzees and corresponds to asymmetries in the brain that are believed to be related to language—a key component of modern human behavior. Yet studies of skull shape in Neandertal specimens representing a range of developmental stages indicate that the Neandertals attained their large brain size through a different developmental pathway than that of H. sapiens. Although Neandertal brains started off growing like modern brains in the womb, they diverged from the modern growth pattern after birth, during a critical window for cognitive development.

Those developmental differences may have deep evolutionary roots. An analysis of some 17 skulls dated to 430,000 years ago from the fossil site of Sima de los Huesos, in the Atapuerca Mountains in northern Spain, has shown that members of the population there, believed to have been Neandertal precursors, had smaller brains than later members of the lineage. The finding suggests that Neandertals did not inherit their large brain size from the last common ancestor of Neandertals and modern humans; instead the two species underwent a parallel brain expansion later in their evolution. Although Neandertal brains ended up approximately as large as ours, their independent evolution would have left plenty of opportunities for the emergence of brain differences apart from size, such as those affecting connectivity.

Genetic Hints

Glimpses of some of those differences have come from DNA analyses. Since the publication of a draft of the Neandertal genome in 2010, geneticists have been mining ancient DNA to see how Neandertals and H. sapiens compare. Intriguingly, the Neandertals turn out to have carried a very similar variant we have of a gene called FOXP2 that is thought to play a role in speech and language in humans. But other parts of the Neandertal genome appear to contrast with ours in significant ways. For one thing, Neandertals seem to have carried different versions of other genes involved in language, including CNTNAP2. Further, of the 87 genes in modern humans that differ significantly from their counterparts in Neandertals and another archaic hominin group, the Denisovans, several are involved in brain development and function.

Differences in the genetic codes of Neandertals and modern humans are not the whole story, however. The switching on and off of genes could have distinguished moderns from Neandertals, too, so that the groups differed in how robustly and under what circumstances they produced the substances encoded by their genes. Indeed, FOXP2 itself appears to have been expressed differently in Neandertals than in H. sapiens, even though the protein it made was the same. Scientists have begun studying gene regulation in Neandertals and other extinct humans by examining the patterns of chemical tags known as methyl groups in ancient genomes. These tags are known to influence gene activity.

But whether or not differences in DNA sequences and gene activity translate to differences in cognition is the big question. To that end, intriguing clues have emerged from studies of people today who carry a small percentage of Neandertal DNA as a result of long-ago interbreeding between Neandertals and H. sapiens.

Geneticist John Blangero of the Texas Biomedical Research Institute runs a long-term study of extended families in San Antonio aimed at finding genes involved in complex diseases such as diabetes. In recent years he and his colleagues had begun looking at brain structure and function in the study participants. A biological anthropologist by training, Blangero started at one point to wonder how he could use living humans to answer such questions as what cognitive abilities Neandertals had.

A plan began to take shape. Over the course of their disease research, Blangero and his team had obtained whole-genome sequences and MRI scans of the brains of hundreds of patients. And they had developed a statistical method to gauge the effects of certain disease-linked gene variants on observable traits. Blangero realized that with the aid of their statistical tool, they could use the Neandertal genomes and his group's genetic and MRI data from living people to estimate the effects of the full complement of Neandertal genetic variants—the so-called polygenotype—on traits related to cognition.

Their results suggest that several key brain regions were smaller in Neandertals than in modern humans, including the gray matter surface area (which helps to process information in the brain), Broca's area (which seems to be involved in language) and the amygdala (which controls emotions and motivation). The findings also indicate that Neandertals would have had less white matter, translating to reduced brain connectivity. And other traits would have compromised their ability to learn and remember words. “Neandertals were almost certainly less cognitively adept,” asserts Blangero, who presented his preliminary findings at the annual meeting of the American Association of Physical Anthropologists in Calgary last April. “I'm willing to bet on that one.”

Of course, without living Neandertals around today, Blangero cannot conduct cognitive assessments that would confirm or refute his inference. But there is, in theory, another way to put his hunch to the test. It would be possible, using existing technology, to study Neandertal brain cell function by genetically modifying modern human cells to have Neandertal DNA sequences, programming them to become neurons and observing the Neandertalized cells in petri dishes. Scientists could then examine the abilities of the neurons to conduct electrical impulses, to migrate to different brain regions and to produce projections (neurites) that aid in cell communication, for instance. Blangero notes that although there are ethical issues to consider where the creation of Neandertal cells is concerned, such work might actually help researchers identify genes involved in modern human brain disorders if the genetic changes compromise neuron function. Such findings could, in turn, lead to the discovery of new drug targets.

Not everyone is ready to draw conclusions about the Neandertal mind from DNA. John Hawks of the University of Wisconsin–Madison observes that Neandertals may have carried gene variants that affected their brain function but that have no counterparts in people today for comparison. He notes that if one were to predict Neandertal skin color based on the genes they share with modern humans, one would surmise that they had dark skin. Yet scientists now know Neandertals had some genes no longer in circulation that probably lightened their skin. But a bigger problem with attempting to suss out how Neandertal brains worked from their genes, Hawks says, is that for the most part researchers do not know how genes affect thought in our own kind. “We know next to nothing about Neandertal cognition from genetics because we know next to nothing about [modern] human cognition from genetics,” he asserts.

Archaeological Insights

Given the limitations of the fossil anatomy and the fact that ancient DNA research is still in its infancy, many researchers say the clearest window on the Neandertal mind is the cultural record these extinct humans left behind. For a long time, that record did not paint a particularly flattering picture of our vanished cousins. Early modern Europeans left behind elegant art, complex tools and remainders of meals attesting to an ability to exploit a wide variety of animals and plants that enabled them to adapt to new environments and shifting climate. Neandertals, in contrast, seemed to lack art and other symbolic remains; their tools were comparatively simple; and they appeared to have had a foraging strategy narrowly focused on large game. Stuck in their ways, the thinking went, the Neandertals simply could not adapt to deteriorating climate conditions and competition from the invading moderns.

In the 1990s, however, archaeologists began to find evidence contradicting that scenario—namely, a handful of decorative items and advanced tools attributed to Neandertals. Ever since, researchers have been at loggerheads over whether these items are Neandertal inventions as claimed; doubt has arisen because the items date to the end of the Neandertal dynasty, by which time H. sapiens was in the area, too. (Anatomically modern humans appear to have reached Europe by around 44,000 to 41,500 years ago, hundreds of thousands of years after Neandertals settled there.) Some skeptics think that H. sapiens made the sophisticated artifacts, which later got mixed in with the Neandertal remains. Alternatively, they offer, Neandertals may have copied the ingenious moderns or stolen their goods.

But that position is becoming harder to uphold in the face of a raft of discoveries over the past few years that evince Neandertal savvy prior to the spread of anatomically modern humans throughout Europe. “There's been a real sea change. Every month brings something new and surprising that Neandertals did,” observes David Frayer of the University of Kansas. “And the new evidence is always that they were more sophisticated, not hicks.”

Some of the most surprising discoveries reveal aesthetics and abstract thought in Neandertal cultures that predated the arrival of H. sapiens. These finds include the engraving and signs of feather use from Gorham's Cave. In fact, artifacts of this nature have turned up at archaeological sites across Europe. At the Grotta di Fumane in Italy's Veneto region, archaeologists found signs of feather use and a fossil snail shell collected from at least 100 kilometers away that had been stained red, suspended on a string and worn as a pendant at least 47,600 years ago. Cueva de los Aviones and Cueva Antón in southeastern Spain have also yielded seashells bearing traces of pigment. Some seem to have served as cups for mixing and holding red, yellow and sparkly black pigments that may have been cosmetics; others bear holes indicating that they were worn as jewelry. The modified shells date to as many as 50,000 years ago.

Other Neandertal leavings indicate that their yen for decorating reaches back further still. Sites in France and Italy document a tradition of harvesting eagle talons that spans from 90,000 to 40,000 years ago. Cut marks on the bones show that the Neandertals focused their efforts on obtaining the claws, not the flesh. This finding led investigators to conclude that the Neandertals exploited the eagles for symbolic reasons—probably to adorn themselves with the impressive talons—rather than dietary ones.

Even older hints of Neandertal aesthetics come from the site of Maastricht-Belvedere in the Netherlands, where archaeologists have found small splatters of red ochre, or iron oxide, in deposits dating to between 250,000 and 200,000 years ago at minimum. The scarlet pigment had been finely ground and mixed into a liquid that then dripped onto the ground. Researchers cannot know for sure what those Neandertals were doing with the red liquid, but painting is one obvious possibility. Indeed, when red ochre turns up at early modern human sites, investigators assume that it was used for decorative purposes.

In addition to rendering a far more resplendent portrait of our much maligned cousins, these new discoveries provide crucial insights into the Neandertal mind. Archaeologists have long considered art, including body decoration, to be a key indicator of modern cognitive abilities because it means that the makers had the capacity to conceive of something in the abstract and to convey that information in symbols. Symbolic thinking underpins our ability to communicate via language—one of the defining traits of modern humans and one that is seen as critical to our success as a species. If Neandertals thought symbolically, as they appear to have done, then they probably had language, too. In fact, abstract thought may have dawned in the human lineage even before the last common ancestor of Neandertals and H. sapiens: in December researchers unveiled a mussel shell from Indonesia that they contend was engraved with a geometric pattern by a more primitive ancestor, Homo erectus, around 500,000 years ago.

Symbolic thought is not the only component of behavior believed to have helped H. sapiens get ahead, however. The manufacture of tools with specialized uses is another element, one that Neandertals appear to have mastered as well. In 2013 Marie Soressi of Leiden University in the Netherlands and her collaborators announced their discovery of bone tools known as lissoirs—implements that leather workers today use to render animal hides more pliable, lustrous and impermeable to the elements—at two Neandertal sites in the Dordogne region of France dating to between 53,000 and 41,000 years ago. Judging from the wear marks on the artifacts, Neandertals used them for the same purpose. The Neandertals made the lissoirs from deer ribs, shaping the end of the bone that attaches to the sternum to form a rounded tip. To wield the tool, they pressed the tip into a dry hide at an angle and pushed it across the surface repeatedly, smoothing and softening the skin.

Fresh evidence of Neandertal ingenuity has also come from the site of Abri du Maras in southern France, which sheltered Neandertals around 90,000 years ago. Microscopic analyses of stone tools from the site, conducted by Bruce Hardy of Kenyon College and his colleagues, revealed traces of all manner of activities once thought to be beyond the ken of the species. For instance, the team found remnants of twisted plant fibers that would have been used for making string or cords, which then could have been fashioned into nets, traps and bags. Traces of wood turned up as well, suggesting that the Neandertals crafted tools from that material.

Residue analysis additionally gives the lie to the notion that Neandertals were perilously picky eaters. Studies of the chemical makeup of their teeth, along with analyses of animal remains from Neandertal sites, have suggested that Neandertals relied heavily on large, dangerous prey such as mammoth and bison rather than an array of animals depending on availability, as anatomically modern humans did. The Abri du Maras Neandertals apparently exploited a veritable menagerie of creatures, including small, fast animals such as rabbits and fish—all species previously thought to be out of reach for Neandertals, with their low-tech gear.

Some scholars have argued that an ability to live partly on plant foods gave H. sapiens an edge over Neandertals, allowing them to reap more sustenance from the same area of land. (Subsisting on plants is trickier for humans than for other primates because our big brains demand a lot of calories, and yet our small guts are poorly suited to digesting large quantities of raw roughage—a combination that requires intimate knowledge of plant foods and how to prepare them.) But the Abri du Maras Neandertals gathered edible plants, including parsnip and burdock, as well as edible mushrooms. And they were not alone.

According to studies led by Amanda Henry of the Max Planck Institute for Evolutionary Anthropology in Leipzig, Germany, Neandertals across a broad swath of Eurasia—from Iraq to Belgium—ate a variety of plants. Examining the tartar in Neandertal teeth and residues on stone tools, she determined that Neandertals consumed species closely related to modern wheat and barley, cooking them to make them palatable. She also found bits of starch from tubers and telltale components of date palms. The similarities to findings from early modern human sites were striking. “Any way we broke up the data, there were no significant differences between the groups,” Henry remarks. “The evidence we have now does not suggest that the earliest modern humans in Eurasia were better at accessing plant foods.”

A Long Farewell

If Neandertals actually behaved in ways once thought to distinguish anatomically modern humans and fuel their rise to world domination, that likeness makes their decline and eventual extinction all the more puzzling. Why did they die out while H. sapiens survived? One theory is that moderns had a bigger tool kit that may have boosted their foraging returns. Modern humans evolved in Africa, where their population size was larger than that of Neandertals, Henry explains. With more mouths to feed, preferred resources such as easy game would have declined, and the moderns would have had to develop new tools to obtain other kinds of food. When they brought this cutting-edge technology with them out of Africa and into Eurasia, they were able to exploit that environment more effectively than the resident Neandertals could. In other words, moderns honed their survival skills under more competitive circumstances than Neandertals had faced and thus entered Neandertal territory with an advantage over the incumbents.

Not only did the large population size of H. sapiens spur innovation, but it helped to keep new traditions alive rather than letting them fizzle out with the last member of a small, isolated group. The bigger, more connected membership of H. sapiens “increasingly provided a more efficient ratchet effect to maintain and build on knowledge compared with earlier humans, including the Neandertals,” offers Chris Stringer of the Natural History Museum in London. Still, the arrival of moderns did not spell instant doom for Neandertals. The latest attempt to track their decline, carried out by Thomas Higham of Oxford and his colleagues, applied improved dating methods to pinpoint the ages of dozens of Neandertal and early modern European sites from Spain to Russia. The results indicate that the two groups shared the continent for some 2,600 to 5,400 years before the Neandertals finally disappeared, around 39,000 years ago.

That lengthy overlap would have left plenty of time for mating between the two factions. DNA analyses have found that people today who live outside Africa carry an average of least 1.5 to 2.1 percent Neandertal DNA—a legacy from dalliances between Neandertals and anatomically modern humans tens of thousands of years ago, after the latter group began spreading out of Africa.

Maybe, some experts offer, mixing between the smaller Neandertal population and the larger modern one led to the Neandertal's eventual demise by swamping their gene pool. “There were never very many of them, there were people coming in from other areas and mixing with them, and they faded out,” Frayer surmises. “The history of all living forms is that they go extinct,” he adds. “That's not necessarily a sign that they were stupid, or culturally incapable, or adaptively incapable. It's just what happens.”

*Erratum (4/8/15): This sentence was edited after posting. The original erroneously stated Gibraltar is the southernmost tip of the Iberian Peninsula—a distinction that belongs to Cape Tarifa.