Child molestation and rape top the social taboo list, according to a survey of 186 people between the ages of 18 and 47, and smoking marijuana ranks lowest among the 19 choices of forbidden behavior. In the middle—worse than robbing a bank but better than spousal murder—lies incest between brothers and sisters. Given the deleterious genetic impacts of offspring from such mating, some researchers have suggested that there may be an evolved mechanism designed to prevent that from occurring. And now evolutionary psychologist Debra Lieberman of the University of Hawaii–Honolulu believes she may have elicited some of its functions from this simple questionnaire.

Many animals show such "kin radar." By mixing siblings in a litter, for example, scientists have shown that animals that grow up together appear to avoid mating, whether genetically related or not, largely based on recognizing specific smells. Of course, this kind of choreographed mixing cannot be done to humans by researchers, but Lieberman and her co-authors John Tooby and Leda Cosmides of the University of California, Santa Barbara, attempted to take advantage of the inherent variability in human families to mimic (somewhat) such directed mixing.

More than 600 subjects filled out questionnaires on their family histories, then each was also randomly given one of three additional sets of questions concerning either: their moral opposition to incest or their altruistic behavior toward their siblings or their level of disgust when imagining sibling incest. Given that the subjects had experienced a wide variety of sibling situations, the researchers hoped to uncover how a kin detection mechanism in humans might work.

The evolutionary psychologists hypothesize that some form of mental mechanism assesses various cues to come up with an estimate of how related two people are. "The real question is: What are these cues?" Lieberman says. "A potent cue is seeing your mom caring for a newborn. That would have served as a great cue that the infant is a sibling, at least a half sibling." But for younger siblings, who would have no opportunity to make this observation, another cue might be the amount of time spent living with another child/potential sibling. Dubbed the "Westermarck hypothesis"—after the Finnish sociologist who first noted it in a book published in 1889—it posits that children reared together do not often end up being sexually attracted to each other.

Whether subjects directly saw siblings as newborns or simply shared the growing-up process, they both behaved more altruistically toward these siblings and felt a stronger aversion to any sort of sexual contact with them. More intriguingly, when the subject had seen the sibling as a newborn, it did not matter how long the two had lived together, the incest aversion was equally strong. For younger siblings (and the small cohort of older siblings who had not seen their younger brothers and sisters with their mothers), this aversion was not as strong unless they had lived together for at least 14 years. "A high estimate [of kinship] should cause aversion to be upregulated," Lieberman says. "A high estimate should also upregulate motivations to act altruistically."

The aversion was also manifested more strongly in women; "for males and females—given the same kinship estimate—in females it will ratchet up aversion even more," Lieberman says, perhaps because women invest more in the genetic fitness of a particular offspring. On the other hand, men who grew up with only male siblings did not seem to find the idea of incest as inherently reprehensible as those who had grown up with sisters—a fact that Liebermann says proves the incest taboo is not learned from parents.

Of course, human behavior can be much darker than these surveys reveal; some experts have argued that as many as 3.5 million American women have been sexually abused by a sibling, though precise statistics are hard to come by due to the shame involved. But Lieberman, for one, is confident that this newly elucidated kin detection system must be malfunctioning in those cases: "There is evidence that cues for kinship are absent in cases where you have sexual abuse happening." The same explanation—in addition to cultural forces—holds for historical examples of incest, she argues, and childhood "exploration" doesn't count. "One of the design features is that individuals who are fertile and genetically related should avoid each other," she says. "Young siblings playing doctor doesn't necessarily contradict."

Ultimately, Lieberman and her co-authors plan to continue to test this potential mechanism, starting with people who do not have the ability to smell. "Another interesting thing to look at is what's going on in blind people or those who can't detect faces," she adds. If they also share these kin detection mechanisms, these evolutionary psychologists may have elucidated how our minds evolved specific mechanisms to keep us from breaking the family taboo--and why most cultures so strongly forbid it.