Dashing for a train in a busy station at rush hour, I picked out a face in the crowd—the familiar configuration of features, the laugh lines and the mole above the right eye. I immediately knew the distinctive visage belonged to my former classmate, Robert.

Most of us are highly skilled at recognizing faces, even though they all have similar features arranged in roughly the same configuration: two eyes separated by a standard-issue nose, along with a mouth, chin and cheeks. We are similarly adept at reading facial expressions to intuit a person’s mood and at extracting information about an individual’s sex, age and direction of gaze. We do this reading within a fraction of a second, an ability that is critically important for normal social interactions.

Human perception of faces exceeds that of other objects and patterns. Can you imagine recognizing a particular Granny Smith apple in someone’s shopping cart that you had just seen in the produce pile? Apples do not appear terribly distinctive to us the way faces do; the same thing is true for trees, cars, butterflies and, well, you name it.

Neuroscientists have long debated the biological basis for human face perception. Because this skill is so critical to communication, many researchers believe that specialized neural hardware has evolved to detect faces. Indeed, back in the 1970s researchers found neurons in a small section of the monkey brain that responded much more strongly to faces than to any other item. Since then, vision scientists have discovered a region in the human visual system that seems similarly sensitive to the human countenance. What is more, people can selectively lose the ability to recognize faces as a result of brain damage or a congenital abnormality [see “Forgetting Faces,” by Thomas Grueter; Scientific American Mind, August/September 2007].

Many psychologists propose that a unique type of visual processing occurs in the region of the brain involved in recognizing faces. Such processing might enable greater perceptual precision and might account for such findings as our spectacular inability to recognize upside-down faces relative to upside-down examples of other objects. Others believe that face-detecting neurons process faces in the same way other brain neurons distinguish objects, except that they are more finely tuned to subtlety because of greater experience with faces. A more contrary group of vision scientists contests the existence of innate face detectors entirely, arguing that practice with faces trains generic object detectors to respond to the human countenance.

Beyond satisfying our curiosity, a better understanding of human face perception might help doctors diagnose and treat disorders such as autism, in which face perception is seriously impaired. It could also aid the quest to develop robotic devices able to tell one person from another by their facial characteristics.

The Upside-Down Effect
The idea that face perception might involve unique neural processes first emerged in the late 1960s, when psychologist Robert K. Yin, then at  the Massachusetts Institute of Technology, compared the ability of 70 students to recognize photographs or drawings of faces with their ability to recognize airplanes, houses and cartoon figures without distinct faces. The students identified the faces more often than the other objects as long as the photographs were right side up. They found all the images more difficult to recognize upside down, but inverted faces were especially hard to discern as compared with the upturned images of the other objects.

Based on this so-called face-inversion effect, Yin proposed that recognizing faces requires some type of visual processing in the brain distinct from that used for perceiving other objects and patterns. In particular, he speculated that face perception may be more holistic—or, all at once—than that of objects, which the brain is thought to perceive from their component shapes.

In the conventional account of visual perception, light detectors at the back of the eye, in the retina, respond most vigorously to spots of light. Signals from groups of these cells eventually coalesce in the primary visual cortex (V1) at the back of the brain, where neurons react best to lines or edges. Signals from those neurons combine to assemble ever more complex shapes as they travel up the hierarchy of visual areas, from V2 through V4 and, finally, to the inferior temporal cortex, where cells are tuned to the perception of complex objects, such as faces, birds and cars.

Such shape-based processing may work reasonably well for most inverted objects. But inverting a face, Yin surmised, might preferentially disrupt a holistic processing that operates only for faces.

Meanwhile other researchers were entertaining alternative explanations for the uniqueness of face perception. Some suggested that instead of processing faces holistically, the brain dissects the human countenance in two steps, by first recognizing its features and then computing their configuration. The face-inversion effect might thus arise from a failure to process the configuration of inverted faces, leaving features as the only guide to the uniqueness of a face.

Psychologist Helmut Leder of the University of Vienna has demonstrated that the spatial characteristics of a face—say, the distance between the eyes and that between the nose and mouth—are important for face recognition and are also very sensitive to orientation. In a 1998 study, for example, Leder and psychologist Vicki Bruce of the University of Edinburgh doctored pictures of faces to alter just their features or the spatial relations among their features. Both types of change made the faces equally more distinctive to viewers and easier for them to recognize than the original face was. But when the faces were upside down, those with unusual feature relations proved far less distinctive or familiar than the faces with touched-up features. Leder and Bruce concluded that face perception involves processing both the individual features and their configuration but that inverting a face preferentially disrupts the latter.

Further evidence suggests that the configuration idea may explain the inversion effect better than the holistic-perception hypothesis does. In 2000 Leder and Bruce reported asking subjects to identify faces either by unique combinations of features, such as eye and hair color, or by distinctive relations between features. As expected, inverting the faces made the ones defined by unusual feature relations much harder to identify than those with distinguishing features. But surprisingly, the faces with odd configurations were also harder to identify upside down than were faces with both distinctive attributes, bolstering the configuration theory over the holistic explanation for the inversion effect.

In 2006 Leder, along with University of Vienna psychologist Claus-Christian Carbon and their colleagues, published work showing that patients with face blindness have the most difficulty with a face-matching task when the faces differ only by their features’ spatial relations. Thus, problems sorting out the configuration of facial features may also explain some pathological deficits in face recognition.

Face Space
Meanwhile researchers had fingered the place in the human brain where such sorting may take place. In 1997 psychologist Nancy Kanwisher, now at M.I.T., and her colleagues used functional magnetic resonance imaging (fMRI) to scan the brains of 15 people while they viewed intact and scrambled faces, full-front views of faces and houses, or three-quarter views of faces and images of human hands. In each case, a blueberry-size region they dubbed the fusiform face area (FFA), located in the fusiform gyrus, reacted more strongly to the intact face stimuli.

Not all scientists are convinced that the FFA homes in on feature arrangements. Yet another theory, first proposed in 1991 by psychologist Tim Valentine, now at Goldsmiths College in London, is that face perception revolves around the representation of a prototype face, against which the brain compares all other visages. In the brain, this reference face may be an average of the activity patterns created in response to seeing many different faces, suggests cognitive scientist Martin A. Giese of the University of Tbingen in Germany. Some scientists visualize a multidimensional face space, which contains the average of all faces at its center and individual faces radiating out from the origin as a function of their distinctiveness. This picture jibes with the observation that exaggerating features, as is done in caricatures, makes faces easier to recognize.

Giese, along with neurophysiologists David A. Leopold, now at the National Institute of Mental Health, and Igor V. Bondar of the Institute of Higher Nervous Activity and Neurophysiology in Moscow, tested the face-space hypothesis in the visual system of rhesus monkeys, whose ability to recognize faces is very similar to our own. They created an “average” human face by merging the characteristics of a large number of human faces and then constructed caricatures based on that norm. They showed these faces to monkeys while measuring the activity of neurons in the inferior temporal cortex, where their face-detecting cells reside.

In 2006 they reported that the average face elicited relatively low levels of activity from the face neurons and that the neuronal responses became increasingly vigorous as the caricatures became more and more distinctive. “Cells that signal deviations from the facial norm react strongly to small variations in the shape of the face,” Giese says. “This [mechanism] makes it possible for us to recognize minimal differences with a limited number of neurons.” It also may explain why changes in facial expression have to be learned only once and not relearned for each new face.

Not everyone is convinced, however, that such findings prove the brain uses a norm-based system for processing faces. For example, computational neuroscientist Maximilian Riesenhuber of Georgetown University says the results may instead reflect the general tendency of neurons to “adapt to a facial norm that is shown frequently and then subsequently respond to it less strongly,” a tendency that is not specific to face recognition.

Shaping Up
Indeed, Riesenhuber rejects the idea that seeing faces requires any such special computation by the brain. Instead, he says, face perception operates by the same rules that object perception does. He and his colleagues created a computer model of standard shape-based visual processing and showed that it could account for the human forte in perceiving faces, along with the extreme preference for upright versions, with one additional ingredient: expertise.

Based on classical visual theory, Riesenhuber’s simulation represents objects as conglomerations of component shapes. Neurons detecting, say, spots or edges feed information to cells that respond to more complex patterns until eventually cells respond to whole objects. Cells in defined regions of the brain react to different classes of objects, and within each area various objects excite different cells—the proposed biological basis of a person’s ability to tell objects apart.

The more neurons devoted to a class of objects, the more distinctions they can make among objects within that class. Thus, when a person develops expertise at recognizing, say, butterflies or cars, Riesenhuber reasons, the brain recruits more neurons to enable finer discriminations among them. “Faces comprise one object class that everyone is expert in,” Riesenhuber explains. So in his model he finessed face perception by devoting a large number of neurons to it: in different scenarios, groups of 70 to 190 neurons in the virtual visual area V4 supply information to 180 individual face units, each tuned to a different face.

To determine how well these imitation neurons could tell faces apart, Riesenhuber and his colleagues fed them digital portraits that differed from an original in a stepwise fashion, from one to 10 “morph steps.” As they reported in 2006, they found that face units receiving a greater number of inputs were more finely tuned to a specific face than were those with fewer inputs; they responded much less vigorously even to faces that were very similar to their “favorite” face. And the more discriminating the face unit, the less it responded to an inverted version of that face, providing an explanation for the inversion effect. “Our model is the first to account for the behavioral data in a quantitative fashion,” Riesenhuber claims.

To ascertain whether the brain actually sees faces this way, Riesenhuber’s team scanned the brains of 13 people while they looked at the morphed portraits. If face cells are simply highly selective shape detectors as the model suggests, then just a small difference in a face should excite a distinct set of neurons in the FFA. Indeed, faces differing by just one morph step activated separate, but overlapping, sets of cells. As the portrait pairs became increasingly dissimilar, so, too, did the responsive cell groups, until at 10 morph steps apart the faces excited totally separate cell clusters. “We have different groups of neurons that respond to different faces,” Riesenhuber concludes. “This enables us not only to distinguish similar faces but to remember new ones more easily.”

People are not born with the ability to make such fine distinctions, Riesenhuber says. Children do not achieve adultlike proficiency at recognizing faces until about age 14, studies suggest. Thus, although innate neural hardware may exist for recognizing faces, experience looking at the human countenance also very likely plays a role in the maturation of the brain’s face areas. Riesenhuber and others believe this process involves the recruitment of additional finely tuned cells.

Stanford University psychologist Kalanit Grill-Spector and her colleagues have now garnered anatomical evidence for that theory. These researchers used fMRI to compare the size of the FFA, among other brain areas involved in object perception, of adults and children. They reported in 2007 that the FFA was considerably larger in adults and that this expansion was correlated with a better memory for faces.

Expert Eyes
People can acquire visual expertise for other objects, of course, and some evidence indicates that such knowledge can produce some of the same perceptual peculiarities that people exhibit with faces, lending support to the view that face and object perception are not so different after all. Back in 1986, for example, M.I.T. psychologists Rhea Diamond and Susan Carey reported that they found an inversion effect for dog faces among dog experts—in this case, experienced jurors of canine beauty contests. In their experiments, the jurors could no longer recognize the breed of a dog when the dog’s photograph was upside down.

Anatomically, some studies show that the basis for such specialized acuity develops in brain areas near, but separate from, the FFA, leaving intact the concept of dedicated neural real estate for faces even if the visual system detects them similarly. In 2004 psychologist Gillian Rhodes of the University of Western Australia and her colleagues pointed to a brain region in butterfly experts that was specialized for parsing butterflies. The researchers found that the neurons that responded best to views of these winged insects were near, but largely separate from, the cells that responded vigorously when the Lepidoptera connoisseurs viewed human faces. “You have learning for butterfly experts in a brain region that is very close to the neurons that like faces,” Riesenhuber comments.

In a 2007 study Riesenhuber and his colleagues documented the biological effect of such visual learning in people with expertise in looking at cars. They determined that a clustered group of neurons in the so-called lateral occipital cortex became more selective for different types of cars after the scientists trained study subjects to recognize cars. Such findings indicate that the brain does use largely separate populations of neurons when learning to distinguish among members of different object classes and that the FFA is the cerebral spot for faces.

That idea remains controversial, however. Other work rebuts the postulate that neurons in the FFA are faithful to faces, instead suggesting that they can switch allegiance to other objects or patterns for which a person has developed expertise. In the late 1990s psychologist Isabel Gauthier, then at Yale University, and her colleagues detected elevated activity in the FFA of test subjects who had been trained to recognize bizarre constructions they called greebles, which vaguely resemble bird heads.

Looking at greebles elicited far less activity in the FFA of people who had no previous exposure to them. What is more, as with faces, the FFA was less active in the greeble experts when they were viewing inverted, as opposed to upright, greebles. Gauthier, now at Vanderbilt University, concludes that the FFA becomes stimulated when a person has to identify a particular item within a group of similar items regardless of the type of object.

But even the view that faces must share their place in the brain does not diminish the wonder of our extraordinary ability to decode them nor their importance in our lives. As 18th-century physicist Georg Christoph Lichtenberg once said: “The most entertaining surface on the face of the earth is that of the human face.”