For as long as he can remember, Bryan Alvarez has thought his mother resembled a Mark Rothko painting. The likeness is not just a metaphor he conjured up one day. Whenever he conceives of her name, Marla, he literally sees, in his mind's eye, blocks of colors, each one blending into the next—grainy, brick red for the M, bright, blood red for the A, eggplant purple for the R, plum purple for the L and red again for the final A.
Growing up, Alvarez never thought it unusual that letters have inherent colors. He was in high school when he learned that his peers did not perceive the world as he did. Alvarez has a condition called synesthesia, in which otherwise normal people feel shapes when they taste foods, smell odors when they hear musical notes or see colors when they read words.
History is littered with accounts of equally strange departures from what is commonly thought of as ordinary perception—blinded soldiers who can dodge bullets, patients who can sense light without eyes and people who can navigate obstacles using sound. For centuries such stories were considered unexplained curiosities, magical gifts or neurological anomalies—exceptions to the basic rules of perception. Only in the past few years have neuroscientists begun to suspect that we all may share some of the mechanisms underlying these conditions.
Until recently, perception was largely viewed as the handiwork of distinct senses. Greek philosopher Aristotle allegedly first classified the famous five: vision, hearing, taste, smell and touch. Over the years researchers have added many more to the list, including senses for balance, pain, time, temperature, limb positioning, and the ability to register hunger and thirst. Brain-imaging studies in the past two decades have helped researchers divvy up the senses further still, identifying neural pathways for processing numbers and letters, colors, shapes and faces. But this old model of perception is beginning to change.
“We can no longer view the brain as a bunch of specialized compartments that don't interact much,” says psychologist Ladan Shams, who studies multisensory perception at the University of California, Los Angeles. Neuroscientists are discovering that our sensory systems are much more interconnected and widespread in the brain than previously thought. Vision is not just about seeing. Hearing is not just about listening. Even in ordinary circumstances, the ways our senses can inform and compensate for one another may seem exotic or even superhuman.
The Eye's Secret Back Door
Most evidence supporting a splintered view of perception comes from studies of vision. Scientists have long known that visual information travels from the eyes through the thalamus, a relay station located above the brain stem, to the visual cortex, where it gets packaged into the colored, textured, three-dimensional scenes we perceive as sight. Even so, researchers are now finding that some visual data are deployed for uses that have little to do with ordinary sight, such as keeping track of time and controlling movement subconsciously. In fact, the sensory mechanisms may be altogether different, starting with tiny, little-known light sensors in the retina.
Since the mid-1800s anatomists have known about the eye's rods and cones, and for nearly two centuries scientists believed these photoreceptors were the only cells in the human body that could detect light. In the early 1990s, however, a young researcher named Russell Foster, now at the University of Oxford, started sharing his suspicions that an unidentified type of eye cell was also responding to light and firing off very different messages to the brain.
Foster, a circadian neuroscientist, knew that the mammalian brain relies on light not just to form images but also to set its internal clock. He was aware, for example, that mice can adjust their behavior to synchronize with a new day-night cycle, just as humans do after traveling to a different time zone. Mice without eyes, however, cannot. Yet when Foster studied mutant mice lacking rods and cones, he found they could reset their clock just fine. “There had to be some other weird photoreceptor residing in the eye,” he says. “But what the bloody hell was it?”
While Foster hunted in vain for the elusive photoreceptor, one of his former graduate students made a startling, seemingly unrelated discovery. Ignacio Provencio, now at the University of Virginia, had identified a protein that makes some skin cells in frogs darken in the presence of light. He called the protein melanopsin. He searched for it in other frog tissues and, to his surprise, uncovered it in cells of the retina. Remarkably, those cells were neither rods nor cones. “This immediately set off a lightbulb,” Provencio remembers. “I thought, Aha! We might have finally identified those mythical photoreceptors we'd been looking for a decade before.”
Sure enough, Provencio observed in the retinas of mice and humans that a small percentage of ganglion cells—which typically carry signals from rods and cones along the optic nerve into the brain—contained melanopsin. Other labs were intrigued by these strange neurons and did many experiments verifying their role in setting the day-night cycles of mice and rats. Then, in 2007, Foster met a woman with a rare genetic disorder that had destroyed her rods and cones but left her ganglion cells intact. He discovered that, like his lab mice, she could adjust her sleep patterns and could even sense whether a room was bright or dark, although she claimed she could not see any source of light. The evidence was glaring: hidden detectors in our eyes guide our biological clocks—and probably do much more.
By tracing the cells' messaging route into the brain, other research groups learned that their signals diverge not just to the sesame seed–size clump of neurons responsible for our circadian rhythms, called the suprachiasmatic nucleus, but also to brain centers that dilate the pupils, shift the gaze, and even regulate fear and pain. Researchers at Beth Israel Deaconess Medical Center in Boston, a teaching hospital affiliated with Harvard University, found in a study in 2010 that for blind patients who experience migraines, light intensifies the pain only when their ganglion neurons are healthy.
“It's become quite clear that there are many, many aspects of our visual system that are taking place at levels you might call reflexive, automatic or subconscious,” says Brown University neuroscientist David M. Berson. The “funny ganglion cells,” as Berson calls them, are just one way the brain surreptitiously collects visual cues to guide our behavior without our knowing it.
Seeing without Knowing
Vision can bypass conscious perception in another, even stranger way. The phenomenon is most easily observed in people with damage to their primary visual cortex, the brain's main image-processing center. In the spring of 2002, for instance, a Scottish ophthalmologist named Gordon Dutton received a visit from a young secretary who had lost her entire primary visual cortex, leaving her completely blind. As Dutton escorted her into his office, he noticed she swerved to avoid a row of chairs in the hallway.
“You walked around those chairs,” he remarked. “What chairs?” she replied, puzzled. “I know you can't see them,” Dutton reassured her, “but could you walk around them again?” She did and still perplexed, admitted, “I don't know how I did it.” Dutton smiled and said, “That's because your unconscious visual brain did it for you.”
Dutton was sure the secretary had what psychologists call “blindsight.” He had read many accounts dating back to the early 1970s of patients with cortical damage who could not identify objects in parts or all of their visual field. Yet when psychologists asked them to either glance toward an object they claimed they could not see, reach out and touch it, or guess its shape or its color, many of the subjects did so correctly far more often than chance [see “Subconscious Sight,” by Susana Martinez-Conde; Scientific American Mind, April/May 2008].
Researchers today explain blindsight as a kind of information detour in cortically damaged brains. With ordinary vision, the images on a retina first get mapped to neurons in the primary visual cortex, located at the back of the skull. From there the signals diverge into two distinct channels. One path taps into memory to identify objects and forms. The other leads to more evolutionarily ancient parts of the brain, some of which control reflexive movements such as catching a fastball or ducking a punch.
But when the primary visual cortex is damaged, recent imaging studies in monkeys and humans confirm, some visual information can take alternative routes. It arrives at the brain's motor centers without ever passing through the parts of the cortex involved in memory and consciousness. “This explains why you can have patients who are unable to tell you if a line is horizontal or vertical, but they can orient their hand and size their grip just right to grasp a pencil in your hand,” says Melvyn A. Goodale, a neuroscientist at the University of Western Ontario.
Blindsight also appears to play a role in recognizing emotions and triggering mood. In a series of studies begun in 1999 Beatrice de Gelder of Tilburg University in the Netherlands showed that some affected patients could accurately guess whether a face was happy or angry. Moreover, they could sense when body postures were threatening and even flexed their facial muscles and dilated their pupils in response—evidence, de Gelder says, that unconscious visual processing “is transporting the patient into a real emotional mood.” [To learn how our senses interact with moods, see “I Know How You Feel,” by Janina Seubert and Christina Regenbogen, on page 54.]
No one can say for sure whether the pathways underlying blindsight also exist in fully sighted people. De Gelder thinks they do, although they may be much less active. In a study currently in press she and her colleague Marco Tamietto of the University of Turin in Italy compared brain scans of a sighted person and a partially blindsighted patient with just one damaged hemisphere. She found the same neuronal connections between the eye and the emotion centers of the brain in both individuals. Yet the links were much stronger in the blindsighted hemisphere.
“We tend to think of brain damage as a loss of function,” de Gelder says. “But we also have to think about it in terms of gaining functions that were inhibited by certain brain areas before. The human brain is like a very, very big delta: if there is a dam on a major route, then water will flow along the minor routes, and those minor routes will become wider and more functional.”
Goodale and Dutton share this view of the brain. They view blindsight as any mental or physical endowment—a skill that can be trained and put to good use when needed. For example, they recently worked together to teach the blindsighted secretary to read large letters printed on a page. Although she does not consciously perceive the letters, she can trace their outlines with her finger (or imagine tracing them) and recognizes the shapes she draws in the air. “She reads the headlines of the Times that way,” Dutton says. “Very slowly, of course.”
Sonic Vision
Besides rerouting messages from the eyes, the brain is also very good at substituting one sensory input for another, even using the ears to construct remarkably sightlike pictures of the world. By examining individuals who have developed this ability, scientists are beginning to realize that the visual system may not be strictly about vision after all.
The most scientifically studied person who sees with sound is an American man named Daniel Kish. He lost his eyes to cancer when he was a year old. As a toddler, he figured out that if he made sharp, quick clicks with his tongue and listened to their echoes, he could get around his neighborhood pretty well. Really well, in fact. Today he goes dancing, hikes in the dark and frequently rides his bike in city traffic.
Scientists refer to what Kish does as human echolocation, although Kish describes it as “something like seeing the world in dim flashes of light.” The scenes he relates have form, texture, depth and continuity, but they are colorless and their resolution is limited to “about the size of a softball.”
Kish often uses the vocabulary of sight to describe his echolocation, and scans of his brain suggest this is no accident. In a 2011 study led by Lore Thaler, then a researcher in Goodale's lab at Western Ontario, scientists scanned Kish's brain, as well as that of another blind echolocator (along with two control subjects), while they listened to recordings of clicks and their echoes. Thaler also had the subjects listen to the recordings without the echoes. Comparing the images, she found that the visual cortex was active only in the echolocators and only when they listened to the echoes, not other background sounds. The auditory cortex, meanwhile, seemed to play no special role in turning echoes into images.
It is well known that when the eyes are lost or damaged, the vision centers in the brain get recruited for other tasks, such as reading braille. The visual cortex is also active when blind people use sensory substitution devices—head-mounted camcorders that translate images into sounds or tiny electrical pinpricks delivered to the skin, for example. But studies such as Thaler's suggest that the visual brain does not rent out its abandoned neural real estate simply because it is available. Rather it may learn to employ new sensory inputs to perform its usual tasks: calculating spatial relationships and composing scenes.
“We're having to rethink what the visual system really is for,” says Alvaro Pascual-Leone of Beth Israel Deaconess and Harvard Medical School. Even in sighted people, he says, the visual brain may also be wired to use signals from the ears and skin. In one study, for example, he blindfolded volunteers for five days and periodically scanned their brains as they solved puzzles involving hearing or touch. By the fifth day he could observe certain vision centers becoming more engaged, and the volunteers' performance on some tasks improved. Pascual-Leone was amazed by how quickly the changes happened. Five days is not enough time for the brain to grow new circuits. It is enough time, however, to strengthen old connections and put them to use.
Sighted people can teach themselves to echolocate, as a study published in 2011 showed. In it, subjects used clicks to find plastic plates in a simple lab environment, in some cases almost as well as Kish. “Just because we're sighted, it doesn't mean we're not using some component of echolocation,” Pascual-Leone says. “When we're seeing, we're not purely seeing. Our seeing is flavored, among other things, by the input of sounds.”
Mmm … Sounds Like Coffee
Echolocation, of course, is not the only instance of collaboration among the senses. If you have ever eaten a meal with a stuffy nose or watched a ventriloquist throw her voice, you know firsthand that the senses inform one another. In fact, our perceptual systems work together more often than we may realize, in ways that are far from obvious.
“The brain is constantly multiplexing information,” says neuroscientist Edward M. Hubbard of Vanderbilt University. Classic studies show, for example, that watching lip movements changes the way you hear words—a phenomenon known as the McGurk effect—and that your sense of balance can alter what you see. (If you look at a straight line after twirling around in a circle, the line will look tilted.) Similarly, sensing changes in acceleration can make objects appear higher or lower than they actually are. The sensation is common among airplane pilots, who perceive the nose of the plane during takeoff as rising faster than it is.
Smell, too, is closely connected to several senses. “Even the best wine tasters' trained noses will be thrown off if you color a white wine red,” says neuroscientist Daniel W. Wesson of Case Western Reserve University, who recently found evidence for a sound-smell, or “smound,” sense. In a 2010 study he showed that neurons in the olfactory system in mice fired electrical pulses in response to tones as well as odors. More remarkably, some neurons had stronger or weaker responses to odors in the presence of certain sounds, suggesting that sounds fine-tune the perception of smell. [To learn how scents interact with memory, see “Smells Like Old Times,” by Marina Konnikova, on page 58.] For some people the senses are so entwined that experiencing one can invoke another, as is the case with synesthesia. For a synesthete, a sip of spearmint tea might have the texture of a glass column, or the note F-sharp might sound distinctly green. The most common form—the kind Bryan Alvarez has—is known as grapheme-color synesthesia, in which people perceive particular colors for letters or numbers. Most instances of the condition are very likely caused by the cross wiring of brain regions that sit close together but do not normally interact. Why these interplays exist in so many people, however, remains something of a mystery.
One theory builds on the observation that we are all born with far more neuronal connections than we end up with as adults. As our brain develops, the connections we use grow stronger and more active, whereas the rest weaken and can be lost altogether. Infants, therefore, may perceive the world much the way synesthetes do. In a 2011 study, for example, psychologists Katie Wagner and Karen R. Dobkins of the University of California, San Diego, found that babies two to four months old made associations between certain colors and shapes. Eight-month-olds, however, did not, suggesting they had already shed the links among the processing centers for colors and shapes. Because the condition runs in families, a genetic component may determine who retains some of the connections.
There is also evidence that although most people do not consciously see, say, the colors of a Rothko painting when they read a name, they still are inclined to make similar associations as synesthetes do. For example, most people tend to pair high-pitched sounds with light colors, sweet tastes or spiky things and to couple low-pitched sounds with dark colors, sour tastes and round items.
“We think we experience the senses separately,” says Lawrence Rosenblum, a psychologist at the University of California, Riverside. In reality, the brain links and synchronizes sensory information from many sources in ways we cannot consciously observe, giving us extraordinary gifts we never knew we possessed for perceiving the world.
