WHEN YOU first look at the center image in the painting by Salvador Dal reproduced at the left, what do you see? Most people immediately perceive a man's face, eyes gazing skyward and lips pursed under a bushy mustache. But when you look again, the image rearranges itself into a more complex tableau. The man's nose and white mustache become the mobcap and cape of a seated woman. The glimmers in the man's eyes reveal themselves as lights in the windows--or glints on the roofs--of two cottages nestled in darkened hillsides. Shadows on the man's cheek emerge as a child in short pants standing beside the seated woman--both of whom, it is now clear, are looking across a lake at the cottages from a hole in a brick wall, a hole that we once saw as the outline of the man's face.
In 1940, when he rendered Old Age, Adolescence, Infancy (The Three Ages)--which contains three faces--Dal was toying with the capacity of the viewer's mind to interpret two different images from the same set of brushstrokes. More than 50 years later researchers, including my colleagues and me, are using similarly ambiguous visual stimuli to try to identify the brain activity that underlies consciousness. Specifically, we want to know what happens in the brain at the instant when, for example, an observer comprehends that the three faces in Dal's picture are not really faces at all.
Consciousness is a difficult concept to define, much less to study. Neuroscientists have in recent years made impressive progress toward understanding the complex patterns of activity that occur in nerve cells, or neurons, in the brain. Even so, most people, including many scientists, still find the notion that electrochemical discharges in neurons can explain the mind--and in particular consciousness--challenging.
Yet, as the late Nobel laureate Francis Crick of the Salk Institute for Biological Studies in San Diego and Christof Koch of the California Institute of Technology have argued, the problem of consciousness can be broken down into several separate questions, some of which can be subjected to scientific inquiry. For example, rather than worrying about what consciousness is, one can ask: What is the difference between the neural processes that correlate with a particular conscious experience and those that do not?
Now You See It...
THAT IS WHERE ambiguous stimuli come in. Perceptual ambiguity is not a whimsical behavior specific to the organization of the visual system. Rather it tells us something about the organization of the entire brain and its way of making us aware of all sensory information. Take, for instance, the meaningless string of French words pas de lieu Rhne que nous, cited by psychologist William James in 1890. You can read this over and over again without recognizing that it sounds just like the phrase paddle your own canoe. What changes in neural activity occur when the meaningful sentence suddenly reaches consciousness?
In our work with ambiguous visual stimuli, we use images that not only give rise to two distinct perceptions but also instigate a continuous alternation between the two. A familiar example is the Necker cube [see illustration on this page]. This figure is perceived as a three-dimensional cube, but the apparent perspective of the cube appears to shift every few seconds. Obviously, this alternation must correspond to something happening in the brain.
A skeptic might argue that we sometimes perceive a stimulus without being truly conscious of it, as when, for example, we automatically stop at a red light when driving. But the stimuli and the situations that I investigate are actually designed to reach consciousness.
We know that our stimuli reach awareness in human beings, because they can tell us about their experience. But it is not usually possible to study the activity of individual neurons in awake humans, so we perform our experiments with alert monkeys that have been trained to report what they are perceiving by pressing levers or by looking in a particular direction. Monkeys' brains are organized like those of humans, and they respond to such stimuli much as humans do. Consequently, we think the animals are conscious in somewhat the same way as humans are.
We investigate ambiguities that result when two different visual patterns are presented simultaneously to each eye, a phenomenon called binocular rivalry. When people are put in this situation, their brains become aware first of one perception and then the other, in a slowly alternating sequence [see box on opposite page].
In the laboratory, we use stereoscopes to create this effect. Trained monkeys exposed to such visual stimulation report that they, too, experience a perception that changes every few seconds. Our experiments have enabled us to trace neural activity that corresponds to these changing reports.
In the Mind's Eye
STUDIES OF NEURAL ACTIVITY in animals conducted over several decades have established that visual information leaving the eyes ascends through successive stages of a neural data-processing system. Different modules analyze various attributes of the visual field. In general, the type of processing becomes more specialized the farther the information moves along the visual pathway [see illustration on page 8].
At the start of the pathway, images from the retina at the back of each eye are channeled first to a pair of small structures deep in the brain called the lateral geniculate nuclei (LGN). Individual neurons in the LGN can be activated by visual stimulation from either one eye or the other but not both. They respond to any change of brightness or color in a specific region within an area of view known as the receptive field, which varies among neurons.
From the LGN, visual information moves to the primary visual cortex, known as V1, which is at the back of the head. Neurons in V1 behave differently than those in the LGN do. They can usually be activated by either eye, but they are also sensitive to specific attributes, such as the orientation of a contour or the direction of motion of a stimulus placed within their receptive field. Visual information is transmitted from V1 to more than two dozen other distinct cortical regions, called the extrastriate visual areas.
Some information from V1 can be traced as it moves through areas known as V2 and V4 before winding up in regions known as the inferior temporal cortex (ITC), which like all the other structures are bilateral. A large number of investigations, including neurological studies of people who have experienced brain damage, suggest that the ITC is important in perceiving form and recognizing objects. Neurons in V4 are known to respond selectively to aspects of visual stimuli critical to discerning shapes. In the ITC, some neurons behave like V4 cells, but others respond only when entire objects, such as faces, are placed within their very large receptive fields.
Other signals from V1 pass through regions V2, V3 and an area known as V5/MT (the medial temporal cortex) before eventually reaching a part of the brain called the parietal lobe. Most neurons in V5/MT respond strongly to items moving with a specific velocity; we say they are speed- and direction-selective. Neurons in other areas of the parietal lobe respond when an animal pays attention to a stimulus or intends to move toward it.
One surprising observation made in early experiments is that many neurons in these visual pathways, both in V1 and in higher levels of the processing hierarchy, still respond with their characteristic selectivity to visual stimuli even in animals that have been completely anesthetized. Clearly, an animal (or a human) is not conscious of all neural activity.
The observation raises the question of whether awareness is the result of the activation of special brain regions or clusters of neurons. The study of binocular rivalry in alert, trained monkeys allows us to approach that question, at least to some extent. In such experiments, each animal is presented with a variety of visual stimuli, usually patterns or figures projected onto a screen. Monkeys can easily be trained to report accurately what stimulus they perceive by means of rewards of fruit juice [see box on pages 10 and 11].
During the experiment, the scientist uses electrodes to record the activity of neurons in the visual-processing pathway. Neurons vary markedly in their responsiveness when identical stimuli are presented to both eyes simultaneously. Stimulus pattern A might provoke activity in one neuron, for instance, whereas stimulus pattern B does not.
Once an experimenter has identified an effective and an ineffective stimulus for a given neuron (by presenting the same stimulus to both eyes at once), the two stimuli can be presented so that a different one is seen by each eye. We expect that, like a human in this situation, the monkey will become aware of the two stimuli in an alternating sequence. And, indeed, that is what the monkeys tell us by their responses when we present them with such rivalrous pairs of stimuli. By recording from neurons during successive presentations of rivalrous pairs, an experimenter can evaluate which neurons change their activity only when the stimuli change and which neurons alter their rate of firing when the animal reports a changed perception that is not accompanied by a change in the stimuli.
Jeffrey D. Schall, now at Vanderbilt University, and I carried out a version of this experiment in which one eye saw a grating that drifted slowly upward while the other eye saw a downward-moving grating. We recorded from visual area V5/MT. We found that about 43 percent of the cells in this area changed their level of activity when the monkey indicated that its perception had changed from up to down, or vice versa. Most of these cells were in the deepest layers of V5/MT.
The percentage we measured was actually a lower proportion than most scientists would have guessed, because almost all neurons in V5/MT are sensitive to direction of movement. The majority of neurons in V5/MT did behave somewhat like those in V1, remaining active when their preferred stimulus was in view of either eye, whether it was being perceived or not.
There were further surprises. Some 11 percent of the neurons examined were excited when the monkey reported perceiving the more effective stimulus of an upward/downward pair for the neuron in question. But, paradoxically, a similar proportion of neurons was most excited when the most effective stimulus was not perceived--even though it was in clear view of one eye. Other neurons could not be categorized as preferring one stimulus over another.
While we were both at Baylor College of Medicine, David A. Leopold and I studied neurons in parts of the brain known to be important in recognizing objects. (Leopold came with me to the Max Planck Institute [MPI] for Biological Cybernetics in Tbingen, Germany, and in 2004 he moved to the National Institute of Mental Health [NIMH] where he continues research on perception using both physiological techniques and functional magnetic resonance imaging [fMRI]. This neuroimaging technique yields pictures of brain activity by measuring increases in blood flow in specific areas of the brain.) We recorded activity in V4, as well as in V1 and V2, while animals viewed stimuli consisting of lines sloping either to the left or to the right. In V4 the proportion of cells whose activity reflected perception was similar to that which Schall and I had found in V5/MT, around 40 percent. But again, a substantial proportion fired best when their preferred stimulus was not perceived. In V1 and V2, in contrast, fewer than one in 10 of the cells fired exclusively when their more effective stimulus was perceived, and none did so when it was not perceived.
While at MPI, Leopold--together with our students Alexander Maier and Melanie Wilke (now both at the NIMH)--demonstrated that the neural response patterns we found by using stimuli-inducing binocular rivalry can also be obtained when the animals view other types of ambiguous visual patterns (for example, displays of objects with ambiguous three-dimensional appearance), which do not involve any local interocular competition. All in all, our recording of multiple-unit activity in the early extrastriate cortical areas showed that different perceptual states correlate with subtle but significant changes in the activity of large populations of cells.
The pattern of activity was entirely different in the ITC. David L. Sheinberg, now at Brown University, and I recorded from this area after training monkeys to report their perceptions during rivalry between complex visual patterns, such as images of humans, animals and various man-made objects. We found that almost all neurons, about 90 percent, responded vigorously when their preferred pattern was perceived but that their activity was profoundly inhibited when this pattern was not being experienced. So it seems that by the time visual signals reach the ITC, the great majority of neurons are responding in a way that is linked to perception.
In short, most of the neurons in the earlier stages of the visual pathway responded mainly to whether their preferred visual stimulus was in view or not, although a few showed behavior that could be related to changes in the animal's perception. In the later stages of processing, on the other hand, the proportion whose activity reflected the animal's perception increased until it reached 90 percent.
A critic might object that the changing perceptions that monkeys report during binocular rivalry could be caused by the brain suppressing visual information at the start of the visual pathway, first from one eye and then from the other, so that the brain perceives a single image at any given time. If that were happening, changing neural activity and perceptions would simply represent the result of input that had switched from one eye to the other and would not be relevant to visual consciousness in other situations. But experimental evidence shows decisively that input from both eyes is continuously processed in the visual system during binocular rivalry.
We know this because it turns out that in humans, binocular rivalry produces its normal slow alternation of perceptions even if the competing stimuli are switched rapidly--several times per second--between the two eyes. If rivalry were merely a question of which eye the brain is paying attention to, the rivalry phenomenon would vanish when stimuli are switched quickly in this way. (The viewer would see, rather, a rapid alternation of the stimuli.) The observed persistence of slowly changing rivalrous perceptions when stimuli are switched strongly suggests that rivalry occurs because alternate stimulus representations compete in the visual pathway. Binocular rivalry thus affords an opportunity to study how the visual system decides what we see even when both eyes see (almost) the same thing.
A Perceptual Puzzle
WHAT DO THESE FINDINGS reveal about visual awareness? First, they show that we are unaware of a great deal of activity in our brains. We have long known that we are mostly unaware of the activity in the brain that maintains the body in a stable state--one of its evolutionarily most ancient tasks. Our experiments show that we are also unaware of much of the neural activity that generates--at least in part--our conscious experiences.
We can say this because many neurons in our brains respond to stimuli that we are not conscious of. Only a tiny fraction of neurons seem to be plausible candidates for what physiologists call the neural correlate of conscious perception--that is, they respond in a manner that reliably reflects perception.
We can say more. The small number of neurons whose behavior reflects perception are distributed over the entire visual pathway, rather than being part of a single area in the brain. The changes in the activity of such neurons during a perceptual transition are small relative to those observed during the physical alternation of the stimuli. Yet such small changes in the response of a distributed neuronal population result in robust alternating patterns of excitation and inhibition of ITC cells during the perceptual dominance and suppression of a pattern, respectively. Although the ITC clearly has many more neurons that behave this way than those in other regions do, such neurons may be found elsewhere in future experiments. Moreover, other brain regions may be responsible for any decision resulting from whatever stimulus reaches consciousness. Erik D. Lumer and his colleagues at University College London have studied that possibility using fMRI. They showed that in humans the temporal lobe is activated during the conscious experience of a stimulus, as we found in monkeys. But other regions, such as the parietal and the prefrontal cortical areas, are activated precisely at the time at which a subject reports that the stimulus changes.
Further data about the locations of and connections between neurons that correlate with conscious experience will tell us more about how the brain generates awareness. But the findings to date already strongly suggest that visual awareness cannot be thought of as the end product of such a hierarchical series of processing stages. Instead it involves the entire visual pathway as well as the frontal parietal areas, which are involved in higher cognitive processing. The activity of a significant minority of neurons reflects what is consciously seen even in the lowest levels we looked at, V1 and V2; it is only the proportion of active neurons that increases at higher levels in the pathway.
It is not clear whether the activity of neurons in the very early areas is determined by their connections with other neurons in those areas or is the result of top-down, feedback connections emanating from the temporal or parietal lobes. Visual information flows from higher levels down to the lower ones as well as in the opposite direction. Theoretical studies indicate that systems with this kind of feedback can exhibit complicated patterns of behavior, including multiple stable states. Different stable states maintained by top-down feedback may correspond to different states of visual consciousness. Neuroimaging studies with human subjects can afford the use of elaborated behavioral paradigms designed to discriminate such top-down from bottom-up processes, but the fMRI signal is mainly sensitive to the input and local processing in each area. Such local activation may or may not excite the large cells (for instance, the pyramidal neurons of cortex) that represent the area's output and are typically isolated and studied in behaving animals.
One important question is whether the activity of any of the neurons we have identified truly determine an animal's conscious perception. It is, after all, conceivable that these neurons are merely under the control of some other unknown part of the brain that actually determines conscious experience.
Elegant experiments conducted by William T. Newsome and his colleagues at Stanford University suggest that in area V5/MT, at least, neuronal activity can indeed determine directly what a monkey perceives. Newsome first identified neurons that selectively respond to a stimulus moving in a particular direction, then artificially activated them with small electric currents. The monkeys reported perceiving motion corresponding to the artificial activation even when stimuli were not moving in the direction indicated.
It will be interesting to see whether neurons of different types, in the ITC and possibly in lower levels, are also directly implicated in mediating consciousness. If they are, we would expect that stimulating or temporarily inactivating them would change an animal's reported perception during binocular rivalry.
A fuller account of visual awareness will also have to consider results from experiments on other cognitive processes, such as attention or what is termed working memory. Experiments by Robert Desimone and his colleagues at the NIMH reveal a remarkable resemblance between the competitive interactions observed during binocular rivalry and processes implicated in attention. Desimone and his colleagues train monkeys to report when they see stimuli for which they have been given cues in advance. Here, too, many neurons respond in a way that depends on what stimulus the animal expects to see or where it expects to see it. It is of obvious interest to know whether those neurons are the same ones as those firing only when a pattern reaches awareness during binocular rivalry.
The picture of the brain that starts to emerge from these studies is of a system whose processes create states of consciousness in response not only to sensory inputs but also to internal signals representing expectations based on past experiences. In principle, scientists should be able to trace the networks that support these interactions. The task is huge, but our success in identifying neurons that reflect consciousness is a good start.
NIKOS K. LOGOTHETIS is director of the physiology of cognitive processes department at the Max Planck Institute (MPI) for Biological Cybernetics in Tbingen, Germany. He received his Ph.D. in human neurobiology in 1984 from Ludwig-Maximillians University in Munich. He then moved to the brain and cognitive sciences department of the Massachusetts Institute of Technology, and in 1990 he joined the faculty of the division of neuroscience at Baylor College of Medicine. Seven years later he moved to MPI to continue his work on visual perception. His recent research includes the application of functional magnetic resonance imaging (fMRI) to monkeys and measurement of how the fMRI signal relates to neural activity. Since 1992 he has been adjunct professor of neurobiology at the Salk Institute for Biological Studies in San Diego; since 1995, adjunct professor of ophthalmology at Baylor; and since 2002, an associate of the Neurosciences Institute in San Diego and senior visiting fellow at University College London. This year he became a part-time faculty member at the Victoria University of Manchester in England. Logothetis is a recipient of the DeBakey Award for Excellence in Science, the Golden Brain Award of the Minerva Foundation, the 2003 Louis-Jeantet Prize of Medicine and the Zlch Prize for Neuroscience.