Which city is farther north--Portland, Ore., or Portland, Me.? Unless for some reason you recently committed to memory the latitudes of all large U.S. cities, you probably have only a rough map of the country in your head and can call up at best an approximate mental image of their locations. Or perhaps your mental picture is so precise that you know the right answer (Portland, Ore.). This ability to conjure internal images may seem matter-of-fact to you, but from a scientific perspective it is anything but.
How the brain generates and processes mental pictures has been a matter of much debate in the research community. The solution to the problem would illuminate an important facet of our conscious experience [see "The Movie in Your Head," by Christof Koch; Scientific American Mind, Vol. 16, No. 4, 2005]. The core of the issue is an even more fundamental question that has occupied philosophers for millennia: What are thoughts made of, and how are they represented in the brain? As with many other areas of neuroscience in recent years, brain-imaging technology is providing some insights.
Sharp or Fuzzy
Philosophers have pondered the origins and purpose of mental imagery since ancient times. The first person to approach the problem using scientific methods was the renowned anthropologist and statistician Francis Galton, a cousin of Charles Darwin. In 1880 he published in the journal Mind the results of a survey he conducted among numerous colleagues and friends. The task he set was to have all respondents imagine the table at which they had breakfast in the morning. Galton was interested in the number of details his test subjects could recall and precisely how the image was constituted. In other words, how colorful and vivid was it? The results puzzled him: the quality of the images reported by his respondents fluctuated widely, and of all people, his fellow researchers overwhelmingly reported that their internal images were faint, obscure or simply nonexistent.
How could that be? Later investigators noted a methodological flaw in the way Galton posed his research question. One person's mental image simply cannot be compared with anothers. What was missing in his question "How clear is your mental image?" was some kind of uniform reference point.
In 1973 English psychologist David F. Marks finally came up with a tool to find such a guide. Now at City University in London, Marks developed the Vividness of Visual Imagery Questionnaire (VVIQ), which is still the most widely used instrument for this purpose. The VVIQ asks respondents to compare the clarity of their mental imagery with the quality of their perception when actually looking at something. The test subjects are asked to imagine a variety of scenes and then to rank on a five-point scale how vividly they see their mental image. Answers range from 1 ("perfectly clear and as vivid as normal vision") to 5 ("no image at all, you only 'know' that you are thinking of an object"). Using this method, Marks discovered that a very small percentage of people claim to see no mental image at all, whereas somewhat more say they see "almost nothing."
Comparing the quality of mental images is one thing; understanding their foundations in the brain is another. Experts generally agree about verbal (that is, nonpictorial) representations. The current consensus is that such thoughts are made of so-called propositions--that is, links between an object and its attributes. For example, "round (ball)" implies that a ball is generally round. More complex thoughts are generated by linking many propositions to one another.
But when it comes to mental imagery, there is nothing like consensus: descriptionalists argue that internal images can also be represented in the form of such irreducibly simple statements. According to this theory, even mental imagery as complex as a wild windswept landscape is made of propositions, which gives us a subjective "as if" sensation that we are seeing an image.
In contrast, the various theories espoused by pictorialists hold that mental images are thoughts that are actually represented graphically in the brain and not in the form of propositions. In the 1960s researchers, particularly cognitive psychologists, began to claim that mental imagery was a fundamentally different kind of mental representation than other forms.
In 1971 Roger N. Shepard and Jacqueline Metzler of Stanford University had demonstrated that the greater the angle by which an object had to be rotated in the inner eye, the longer it took to rotate it. A plausible conclusion is that objects are represented in image form in the brain--and perhaps even in three-dimensional form. You can test another example yourself. Imagine the capital letter D and rotate it counterclockwise by 90 degrees. Now take the capital letter J and stick it into the middle of the lower edge of the D that you just rotated. What do you see? The most frequent answer is a line drawing of an umbrella. What this means is that people are able to attribute new meanings to objects in their inner eye--just as they can with real images. This would hardly be possible if the imagery consisted simply of individual propositional linkages.
The best-known representative of the pictorialist approach is Harvard University psychologist Stephen M. Kosslyn. His model assumes that visual memories are stored in propositional form or are similarly coded--analogously to the way that digital cameras compress image data to store them. Memory then transfers the data to a system that Kosslyn calls a "visual buffer." Here the propositional information contained in the image is transformed and graphically represented in a manner that is akin to the way digital cameras display images in the viewer.
Exactly what this visual buffer consists of was long unclear. Kosslyn now thinks that he knows what part of the brain may be responsible: nerve cells in the visual cortex whose spatial arrangement mirrors that of the light receptor cells in the retina. According to this idea, a checkerboard pattern of light falling on the retina stimulates cortical neurons that are arranged in a comparable pattern--in other words, the nerve cells graphically make an image of what has been seen. When we see images in our mind's eye, we are actually seeing them; however, the input to our visual center is coming from memory, not the sense organs.
If this theory were correct, the imagery in the inner eye would have to pass from the visual cortex through the usual processing steps necessary for seeing. Experiments using fake maps indicate that this may well be the case. Kosslyn used a drawing of a small island on which he marked various points such as a beach, light tower, tree and elephant. The test subjects were asked to memorize this map, particularly the position of the points and objects. They were then asked to imagine the beach itself and to tell the technician when they saw a mental image. Shortly thereafter, they were given the name of another point. The study subjects were then asked to search their mental map for this new site and to press a button when they had "found" it--that is, had a mental image of it.
Over the course of this experiment and similar tests, Kosslyn repeatedly found that the farther apart these two points were on the fictive map the more time his test subjects needed to perform this action. He concluded that the map must be represented as an image in the brain and that they had to "scan" the image with their inner eye to find the point in question--exactly the same as would occur when looking at a real map. If, on the other hand, knowledge of the position on the map were in propositional form, the speed of response would always be approximately the same, according to Kosslyn.
As impressive as the empirical evidence presented by Kosslyn and other pictorialists may be, their critics, the descriptionalists, are not about to give up. In particular, psychologists Zenon Pylyshyn of Rutgers University and Peter Slezak of the University of New South Wales in Australia have for many years been analyzing critically the experiments of the pictorialists. Pylyshyn has advanced what he calls a "null hypothesis," seeking to prove that the widespread theory of a quasi-pictorial processing of mental imagery is not consistent with the known facts.
He claims that the findings of the pictorialists result from the fact that the test subjects know how actual seeing occurs. Kosslyn and his colleagues merely instruct their test subjects to use this "tacit knowledge" and to treat their own propositional image representations as if they possessed the characteristics of actual images. To prove this assertion, Pylyshyn posed the following experiment. He instructed his test subjects to memorize a map in which the individual points were marked with small lamps, only one of which was on at any given time. They were then asked to throw an imaginary switch, whereupon that light went off, and another one went on--that is, it took no time whatsoever to jump from point to point. When his test subjects roved mentally around the map, the time it took for changes to take place was independent of the distance between any two points.
For Pylyshyn, this is evidence that the map has to be represented propositionally, because although the test subjects were supposedly using their image representations, there was actually no indication that they needed to scan the map in their minds to switch from point to point. According to Pylyshyn, the objects in the inner eye and the things that we actually see obey entirely different laws [see illustration on preceding page]. "Images on the retina/cortex have yet to be interpreted, while mental images are the interpretation," he writes.
Other researchers support this position with case studies of brain-damaged patients such as M.D. After suffering brain damage, M.D. was no longer able to identify objects that he saw in the real world. He continued to be able to imagine them in his inner eye, however, and he could still draw them accurately--although he was then unable to recognize what he had just drawn. Deficits such as this are almost impossible to reconcile with the notion of uniform processing of mental and real images.
Has empirical evidence brought us any closer to a resolution of the 30-year-old debate over how mental images arise? Perhaps. Over the past decade, new imaging technologies have begun to hold out the prospect of closure.
A Look Inside
If Kosslyn is right, visual representation and actual vision must activate the same areas of the brain. And this is exactly what Kosslyn found in 1997. In a study using positron-emission tomography (PET), two thirds of the areas stimulated by visual representation and actual vision corresponded. The primary visual cortex in particular, which Kosslyn considers to be the origin of our mental images, seemed to be activated during the process of imagination.
Still, several other research teams have been unable to replicate Kosslyn's PET results. Isabel Gauthier and her colleagues at Vanderbilt University demonstrated that the centers involved in object recognition show no particular activity when test subjects are asked to rotate geometric bodies in their heads--but there is such activity when they actually look at those objects. Instead areas of the parietal lobe that balance visual information and the spatial position of the body are activated during rotation.
In addition, shouldnt we find it difficult to distinguish between mental and external images if the information takes the same pathway from the visual cortex? Wouldnt these images constantly interfere with one another? And by extension, shouldnt we be able to make mental images more distinct if we minimized that interference? Many people, however, see mental imagery better with their eyes open than when they are shut. This was the finding of psychologist Stuart J. McKelvie of Bishop's University in Quebec, when he subjected the results of a large number of experiments on image representation to a meta-analysis.
One aspect of mental imagery has been more or less neglected in all the discussion: the connection between visual representations and memory. According to Kosslyn, mental snapshots decay relatively quickly. Nevertheless, all of us are aware of counterexamples. Scenes described in certain books can leave extremely vivid visual representations in our minds. Later, sometimes years later, when a film version is made, we may be disappointed because the scene on the screen is nowhere near as intense as the one in our heads.
Imagined details do sometimes intrude on our memories of real events. Numerous studies have been conducted on this phenomenon. Psychologist Elizabeth F. Loftus and others at the University of Washington discovered that after reading a booklet that described recollections of being lost in a supermarket, many subjects suddenly reported recalling how they, too, had gotten lost as children. Before reading, they reported being unable to recall such scenes--afterward, they unknowingly smuggled images from the book into their own memory base.
One thing is certain: the final word on mental imagery has not yet been uttered. Kosslyn and Pylyshyn presented their divergent views in 2003 in the journal Trends in Cognitive Sciences, having at each other with barely disguised animus. The search for experimental proof--a knockout punch--continues unabated.