A form of synesthesia exists in all our brains. For instance, we speak of certain smells of particular liquids--like nail polish--as being sweet, even though we have never tasted them. This might involve the close neural links and cross activations--that is, when one area of the brain affects another--between the areas involved in smell and taste. This would make sense functionally--e.g., fruits are sweet and also smell sweet like acetone. But it also makes sense structurally, because the brain pathways for smell and taste are closely intermingled and they both project signals to the same parts of the frontal cortex during sensory processing.
Here's another example. Consider how, even as infants, we scrunch up our noses and raise our hands when we encounter disgusting smells and tastes. We also use these gestures, as well as the word "disgusting," to describe a person who is morally questionable. Why do we use the same word as for taste? Why not say he is "painful," for instance?
The reason, we suggest, is that there are underlying evolutionary and anatomical constraints at work. In lower vertebrates, certain regions of the frontal lobes have maps for smell and taste. But as mammals became more social, the same maps were later usurped by evolution for social functions such as territorial marking, aggression and sexuality, eventually culminating in mapping a whole new social dimension: morality. Hence, the interchangeable words and facial expressions for olfactory or gustatory disgust and moral disgust.
The article describes one colorblind subject who could experience certain colors only when making synesthetic associations; he could not see them with his normal vision. Could that effect only occur for a colorblind synesthete, or could synesthetes with normal vision also have the experience of seeing exotic colors?
The effect is most obvious and pronounced in the colorblind synesthetes, but occurs in "regular" synesthetes as well. The colors evoked by cross activation in the fusiform gyrus "bypass" earlier stages of color processing in the brain, which may confer an unusual tint to the colors evoked. This is important for understanding the phenomenon of synesthesia, because it suggests that the qualia label--that is, the subjective experience of the color sensation--depends not merely on the final stages of processing but on the total pattern of neural activity, including earlier stages.
If cross activation is the correct explanation for synesthesia, why is it that the condition often only works in one direction? That is, a synesthete may see numbers or letters as colored, but looking at colors doesn't evoke letters or numbers.
This may have something to do with the manner in which certain sensory dimensions like color are represented in brain maps, as opposed to the way in which numbers are mapped; this might confer an inherent bias toward unidirectional activation. If a number evokes a color then there's something in the visual image--the number--that the color can be ascribed to. Free-floating qualia not anchored to anything, however, may not be possible. That is, if a color evokes a number, where would the number be seen and how big would it be? It would have to be free floating, and that may not be possible. We also think of metaphors as arbitrary, but in fact they are constrained by evolution and by neural hardware (see main article). For example, we say "loud shirt" or "sharp" taste, but rarely "red" sound or "bitter" touch.
For synesthetes who associate a number or letter with a color, does the font, or typeface, matter?
The font that evokes the most optimally saturated color is usually the simplest--for example, the clean lines of the typeface Helvetica rather than the ornate Gothic. But we have seen rare examples in which an unusual font was more effective at invoking strong color. We suggest that such fonts might serve as ultranormal stimuli that evoke even higher responses from the neurons involved with graphemes (the physical appearance of letters or numbers) than more prototypical fonts.
What if the number is presented in the "wrong" color?
If the number has a different color than the one the synesthesia evokes--a green 5, instead of the synesthetic red, for example--it takes slightly longer for the synesthete to name the color. The induced color delays the ability to report the real color. This effect, called stroop interference, shows that the color associations are automatic.
Is a given number always linked to the same color across different synesthetes?
No. One synesthete might see 5 as red, another might see that number as green. But the associations are not random either. There's a higher chance that 5 will be red than it will be, say, blue or yellow.
Although such trends have been noticed before, nobody had a good explanation. We suggest that the effect may reflect the manner in which phonemes, or the word sounds, in certain synesthetes are mapped near an area of the brain called the TPO junction in a systematic topographic manner; this in turn would make certain types of cross activation more likely than others. Similarly, graphemes (letter or number shapes) might be mapped in "form space" in the fusiform gyrus in such a way that makes certain colors correspond with color neurons in an area in the back of the brain involved in color processing, called V4.
A systematic search for such correlations has yet to be attempted, and the few that have been undertaken have yielded ambiguous results. But an analogy with the periodic table of elements might be appropriate. Initial attempts to classify elements produced a certain non-random clustering of properties--for example, the alkaline metals versus the halogens. But no rhyme or reason could be discerned until Mendeleev noticed that, when the atoms were arranged according to atomic weights, the properties tended to repeat. When discrepancies emerged, Mendeleev insisted that the empirical data on atomic weights were wrong--and later research has vindicated his view! Indeed, he was even able to predict the existence and properties of missing elements.
We believe its only a matter of time before analogous correlations and patterns emerge for the rules of cross activation in synesthesia. For instance, if you assume that the A-E-I-O-U sequence is arbitrary, it may not initially be obvious why that vowel sequence maps in a non-arbitrary manner to a certain sequence of colors (we have seen some hints of this). But the point is, the sequence may not be arbitrary. It may reflect progressively anterior mouth and tongue articulations in the mouth needed to make those vowels, which in turn might be mapped in a topographically organized phoneme space in the brain.
Are there other odd types of synesthesia?
One strange type of synesthesia, originally noted by Francis Galton, involves what's called the "number line," which runs in families. If asked to visualize numbers, the subject finds that they are arranged in a continuous line extending from one point in the visual field to another remote point--say, from the top left corner to bottom right. The line doesnt have to be straight; it's sometimes curved or convoluted or even doubles back on itself. Usually the earlier numbers are more crowded together on the line and often they are also colored. Such individuals also often have "calendar lines" depicting months of the year or days of the week sequentially. In one of our subjects, the number line is centered around "world centered" coordinates. He can wander around the 3-D landscape of numbers and "inspect" the numbers from novel vantagepoints. We plan to investigate these phenomena using brain imaging studies and by temporary "lesions" (produced by magnetic stimulation) in the brains of volunteers.
Are there neurological disorders that disturb metaphor and synesthesia?
This has not been studied in detail, but in patients who have lesions in an area of the brain called the angular gyrus we have seen disturbances in the ability to process proverbs and with the bouba-kiki effect (the linking of certain sounds with shapes--see main article). There are also hints that patients with right-hemisphere lesions show problems with metaphor. It's possible that their deficits are mainly with spatial metaphors, such as, "He stepped down as director."
Schizophrenics also have difficulties with metaphors, often interpreting them literally. When asked, What does "a stitch in time saves nine" mean?, a schizophrenic might only say, "It's important to add several stitches before the hole becomes too large." Such subjects also have difficulty with abstraction. Ironically, they are very good at making puns--often doing so without intending to be funny. When asked, "What does a man have in common with an elephant?" a schizophrenic may say, "They can both carry a trunk." Both metaphors and puns involve revealing hidden similarities. So why are schizophrenics bad at one and good at the other? Actually, puns are in some ways the opposite of metaphors: a metaphor reveals a deep similarity, whereas a pun is a superficial similarity masquerading a deep one--hence its comic appeal. So perhaps its not all that surprising that punning can survive or even be enhanced while abstraction and metaphor are compromised.
If we are to understand the neural basis of abstract thought and metaphor, we need to study the manner in which they break down, piecemeal, in neurology and psychiatry; no progress can be made by lumping them all together under "dementia," which is the current practice.
What if a synesthete sequentially learns two languages for which the graphemes are different?
In the native language (for example, English), letters (graphemes) are colored. But the graphemes of a second language learned in adulthood (for instance, Japanese) are initially not colored. After a few months to a year , the characters in the second language start assuming entirely sound-based colors--that is, the colors they would be if they were written in roman letters. This is true for both of the Japanese phonetic alphabets, Hiragana and Katakana. However, letters that don't have English phoneme equivalents tend to assume either arbitrary colors or colors of graphemes that physically resemble the English writing.
With the Chinese writing Kanji--in which a symbol or grapheme stands for a whole word rather than a phoneme, or sound--the symbol takes on pronunciation-based colors. For example, for one synesthete subject, the Kanji word for "Love" is pronounced "Ai." For the letters "A" and "I," the subject associates red and black, respectively. For our subject, this particular Kanji is therefore red with a "touch of black." But there are exceptions, where the meaning of the Kanji overrides the pronunciation, giving it the Kanji symbol the color of the word that represents the meaning in English! For example, the word for "west" in Kanji is "Nishi"--which in Roman letters for our subject should have been purple, black, yellow, red-purple and black. (The symbol is the same as for love but has a different meaning when used in a certain context with a different pronunciation.) But this Kanji always looked green because the word sound "west" in English letters would be green, orange, yellow, blue--and the concept of "west" dominates over the Kanji pronunciation at the level where the synesthetic color is evoked. Also, the colors of the Kanjis for numbers are the same as those for the corresponding Arabic numerals rather the colors of the Kanji sounds themselves.
Effects of this sort may help us elucidate how sound, meaning and symbols interact with each other in the human brain to generate more complex linguistic tokens--and how different components of such tokens elicit synesthetic colors at different stages of the brain's processing systems.
Intriguingly, when the English words with very different meanings started with similar letters, our English-Japanese synesthete often confused the words; She found this frustrating because she often confused the words "stop" and "start"!
Conversely, such individuals often report that it was the "color coding" of a new script that helped them learn the script much faster than they would have otherwise. The observation, together with our demonstration of " blindsight" in synesthetes, suggests a new way for treating dyslexia; perhaps adjacent letters in a word, or adjacent words can be made less "confusable" and more legible by coloring them differently. We are presently investigating this possibility.
Does synesthesia throw new light on the philosophical riddle of qualia--that is, the subjective quality of sensations like pain or the experience of seeing a color such as red or green?
The neurons in an area of the brain called V4 (and other color centers) are not that different physically from, say, the neurons in the auditory cortex concerned with hearing, yet why does their activity feel so utterly different? Synesthesia offers an empirical solution to a longstanding riddle of philosophers.
Philosophers have invented a famous thought experiment called "the Mary riddle" to illustrate the problem. Imagine a brilliant neuroscientist named Mary who has been raised since birth in an entirely black-and-white world; she never gets to see and experience actual colors. She has abstract theoretical knowledge about the color vision of other humans, about the physics of light and wavelength, and about all the brain circuits in both herself and others that allow color discrimination. But she doesn't understand the ineffable quality of "redness" or "blueness." Assume for the sake of argument (remember this is a thought experiment) that the color pathways in her brain have not degenerated because of the lack of stimulation.
Now suddenly you show her a red apple. One of three things could happen. First, maybe nothing happens; she says she just sees gray. Second, she might say, "Wow! So this is what people mean by 'red.'" That is, she encounters qualia for the first time. Third, she could experience a form of "blindsight" for color. A red apple and gray apple look the same to her subjectively, but when asked to point to the red apple she does so correctly, even though it lacks subjective qualia.
Which of these three possible outcomes will actually occur? We believe we've learned the answer from a colorblind synesthete subject. Much like the theoretical Mary, our colorblind synesthete volunteer can not see certain hues, because of deficient color receptors. However, when he looks at numbers, his synesthesia enables him to experience colors in his mind that he has never seen in the real world. He calls these "Martian colors." The fact that color cells (and corresponding colors) can activate in his brain helps us answer the philosophical question: we suggest that the same thing will happen to Mary.
What happens when you reduce the contrast of the number?
In one study subject, when we decreased the contrast of a number on a computer screen, he noticed a matching reduction in the saturation of the synesthetically induced color; he did not experience any colors when the contrast was below about 8 to 9 percent, even when the number was still clearly visible. Such sensitivity to the elementary physical parameters defining the written number, or grapheme, also supports our view that the effect is indeed sensory, rather than a phenomenon of memory.
Further, in 2002 we showed that it is the actual physical contrast--not the perceived contrast--that determines the vividness of color evoked. In one test, we showed subjects a gray number that was printed on each of the two sides of a roof made of a shaped, folded white card. (This is called a Mach card illusion.) The card was illuminated from one side (say, the right) so the other side was in shadow. Even though the physical luminance is radically different on the two sloping sides of the card , their perceived reflectance is identical because the brain takes fact this "into account" and subtracts the effects of the bright light on the right. Yet even though the perceived reflectance was identical on both sides the color saturation looked grossly different to our subject because of the difference in actual physical contrast. Conversely, if the Mach card was mentally reversed in depth its perceived reflectance changed. The right side now looked much lighter than the left because the brain says (in effect) "It's in shadow--so its actually much brighter than the physical luminance indicates." The vividness of colors of the letters, however, remained constant--showing that its the actual physical contrast--not the perceived contrast--that determines the strength of color evoked.
Does synesthesia affect unitization, or the ability to mentally group letters into sets?
We presented this sentence to one of our higher synesthetes: "Finished files are the result of years of study combined with the experience of years." We asked her to count the number of F's in it. Normal, non-synesthetes usually detect only three--they don't see the F's in the three "of's," because "of" is a word that occurs at high frequency so it is not processed as a string of letters but as a unit.
Likewise, our synesthete said that she initially saw only three "red" F's in the sentence. But on careful scrutiny, she saw all six F's tinged red. This suggests that the overall phonetic context within which the letter is embedded can influence the nature and extent of cross activation of the areas of the brain involved in synesthesia. We usually think of vision as a one-way hierarchy or analogous to a bucket brigade. But such contextual effects must be based on "top down" influences from the hearing centers feeding back into the visual grapheme center in an area of the brain called the fusiform gyrus.
In this context it might be interesting to see if L and R are seen as having the same colors by Chinese American synesthetes, given their propensity to confuse these two phonemes.
Can synesthetically induced colors influence motion perception?
We began with a small cluster of 2's embedded in a large matrix of 5's on a computer monitor As indicated in our article, normal individuals cannot see the embedded cluster. This whole display was flashed on the screen as the first frame of a movie and followed by frame two in the same location. In frame two the exact positions of the 5's were randomized and the cluster of 2's as a whole was shifted rightward or upward. Normal observers simply saw random incoherent motion, with the 2's and 5's randomly and indiscernibly changing into each other. But when we showed the same display to a synesthete who saw 2's as red and 5's as green, she saw a "red cluster" jumping left and right (or up and down) on a "green background"--the green being the synesthetic color induced by the 5s.
This observation, made in 2002, supports our view that the synesthetic colors are evoked relatively early in the visual processing hierarchy--early enough to actually drive motion detectors in the visual motion areas of the brain.
Is there are clear distinction between "higher" and "lower" synesthetes?
This remains to be seen; the terminology is no more than temporary shorthand. The distribution might be bimodal, or they may just represent different points on a continuum. What's clear is that that even among synesthetes who associate colors with numbers, there are at least two types: those for whom the physical shapes of numbers evoke colors ("lower") and those who experience color for numbers that are abstract, such as a calendar month ("higher"). There are probably more types, depending on what genes are expressed at what stage anatomically in the number-color processing hierarchy.
In "higher" synesthetes, who associate colors with numbers, days of the week or months of the year, is it the numerical sequence (ordinality) alone that determines the color?
We have determined that in many synesthetes, it's indeed the sequence. But in others the whole word seems to simply take on the color of the first letter of the day or month.
Can we explain the type of synesthesia called "number line," in which each number is linked in an orderly fashion to a specific visual location?
One very strange type of synesthesia that runs in families, originally noted by Francis Galton (a cousin of Charles Darwin) and still mysterious to us, involves what is called the "number line." If asked to visualize numbers, the subject finds that they are arranged in a continuous line extending from one point in the visual field to another remote point--say from the top left corner to bottom right. The line doesn't have to be straight. Sometimes it's curved or convoluted or even doubles back on itself. In one of our subjects the number line is centered around "world centered" coordinates. He can wander around the 3-D landscape of numbers and inspect the numbers from novel vantage points. Usually the earlier numbers are more crowded together on the line and often they are also colored. Such individuals also often have "calendar lines" depicting months of the year or days of the week sequentially, phenomena that we plan to investigate using brain-imaging studies and by using magnetic stimulation to create the effect of temporary "lesions" in the brains of volunteers.
A recent experiment we did clearly establishes the objective reality of these number lines. Normally, when a normal, non-synesthete subject is asked, "Is 55 less than 57?" his reaction time is longer than if he is asked," Is 55 less than 95?" This is called the number-distance effect, because the reaction time tends to be shorter when the numbers are more different in value, or distant, on an imagined number line. Its as if the numbers are actually laid out in an orderly manner in an imaginary neural " line" of sorts, so numbers farther away on the line are easier to tell apart (and more quickly discriminable) than numbers that are close. This was shown by Stanislaus Dehaene of the Collège de France in Paris.
But what happens in the case of a synesthete who has a convoluted number line that doubles back on itself so that, say, 95 is actually closer to 55 in mental space than 57? Would the reaction time depend on the synesthetic or on the real numerical distance? We recently tested this possibility in a subject who has a highly convoluted number line. To our amazement, we found that in him there was no regular "real" number line. His reaction time did not vary with numerical distance or ordinality, as it would in a non-synesthete. Indeed, it seemed to vary more with the distance in Cartesian space than the numerical distance. To our knowledge this is the first objective evidence that convoluted number lines do indeed have an "objective" reality in the brain.. We believe such synesthetic number lines may be represented in the vicinity of the TPO junction or in the angular gyrus of the human brain.
It's tempting to speculate that some such complex number lines or "number landscape" may underlie the extraordinary arithmetic skills of autistic savants (who calculate primes with ease) or even mathematical geniuses like Srinivasa Ramanujam or Pierre Fermat. Could savants be "sliding" their number lines in a manner analogous to our use of slide rules? Are geniuses able to discern more subtle patterns of relationships between numbers because, unlike less-gifted people, they have these number landscapes in their minds that they can roam over and "inspect"?
Updated June 26, 2003.
Back to Hearing Colors, Tasting Shapes
