Fossil records can tell us a lot about our evolutionary past: what our ancestors looked like, how they walked, what they ate. But what bits of bone don’t typically reveal is why humans evolved the way we did—why, compared with all other known species, we wound up capable of such complex thought, emotion and behavior.

A team of researchers has now used a novel technique to form a hypothesis on the origins of our rich cognitive abilities. They did so by profiling the chemicals buzzing around our brains. These compounds, known as neurotransmitters, are the signaling molecules responsible for key brain functions. Their research reveals that in comparison with other higher primates, our brains have unique neurotransmitter profiles that probably resulted in our enhanced cognition.

The authors of the new study—a multicenter effort led by Kent State University anthropologists C. Owen Lovejoy and Mary Ann Raghanti and published January 22 in PNAS—began by measuring neurotransmitter levels in brain samples from humans, chimpanzees, gorillas, baboons and monkeys, all of whom had died of natural causes. Specifically, they tested levels in the striatum, a brain region involved in social behaviors and interactions. Compared with the other species tested, humans had markedly increased striatal dopamine activity. Among other functions, dopamine helps drive reward activity and social behaviors. In the striatum in particular it contributes to uniquely human abilities and behaviors like complicated social group formation and, in part, speech and language.

Humans, gorillas and chimps also had elevated striatal serotonin, compared with other primates. Increased serotonin levels in the striatum are known to increase cognitive and social control and also reduce aggression whereas low levels are linked with underdeveloped social skills.

Also in the mix is the neurotransmitter acetylcholine, higher levels of which are associated with aggression. Lovejoy and his colleagues found that gorillas and chimps have much higher levels of acetylcholine than do humans. “The high striatal serotonin shared by humans and great apes likely contributes to the cognitive flexibility required for complex social interactions,” Raghanti says. “The lower acetylcholine in humans corresponds to our decreased aggression, compared to most other apes. It’s a concert really.”

Raghanti and Lovejoy believe the human brain’s neurochemical profile was shaped by natural selection due to the various reproductive and survival benefits it conferred. Our evolving chemical signature, they suggest, allowed us to outcompete other apes and early hominins, referring to the numerous humanlike species that arose after our split with chimpanzees over six million years ago.

The team speculates humans’ elevated striatal dopamine levels in particular would have led to more advanced social behaviors and perhaps monogamy, both of which may have improved our offspring’s survival and benefitted our ancestors. They also feel that by enhancing social behaviors, a “dopamine-dominated striatum” personality type, as they call it, would have led to selection for increased brain size and also language.

Clifford Jolly, an anthropologist at New York University who was not involved in the new research, believes the authors’ hypothesis is credible. “The suggestion that differences in [neurochemical profiles] are correlated with particular ape–human differences in temperament and behavior remains a hypothesis, although a strongly-founded one,” he says. “It’s certainly probable that the described striatal reorganization played an important role in the evolution of distinctive human behavior, especially the striking level of empathy with other humans that appears to be an innate characteristic of the species.”

Previous research suggests our ability to cooperate and exhibit empathy—both thought to be critical to human success—relied in part on the large brains of our hominin ancestors, relative to body size; and that selection against aggression within early human populations allowed us to thrive. This is called the “self-domestication hypothesis.” In planning the new study Raghanti wanted to test whether or not a large brain was necessary for us to self-domesticate. And given its role in modulating social interactions, she and her colleagues decided the striatum would be a logical brain region to test.

Based on the new findings, Raghanti believes it was our chemical signature that came first, which then allowed our brains to balloon. “Our findings provide a mechanism for how our lineage ended up with large brains [in the first place],” she says. And, as she points out, there is anatomical evidence supporting this chronology.

Fossil records show our hominin lineage emerged before our brains began enlarging.  By the time our crania began expanding, our canine teeth—our “fangs”—had already begun shrinking significantly in size. Raghanti says our smaller canines can be seen as “social teeth.” Long used by sparring males vying for female attention, a shortening of lethal protuberance signals increased civility among early hominins. “A reduction in the size of the canine indicates decreased aggression, which would be consistent with the neurochemical profile that we see in modern humans,” she says. Importantly, this could have occurred without any enlargement of the brain.”

To follow up their study, Lovejoy and Raghanti plan to test their idea on deceased apes who have had their behavior monitored throughout life. If their idea holds, more monogamous primates should have higher dopamine levels whereas the more territorial and aggressive among our evolutionary cousins should have elevated acetylcholine.

“Lineage identification [via the fossil record] is interesting to some, but it tells us very little about why humans are so extraordinarily intelligent and social,” Raghanti says. “Brain chemistry can tell us so much more.”