Decoding Brain Scans: Although companies are starting to pursue brain decoding for a few applications, such as market research and lie detection, scientists are far more interested in seeing if these techniques can tell them about the basic principles governing brain organization and how it encodes memories, behavior and emotion Image: Nature
Jack Gallant perches on the edge of a swivel chair in his lab at the University of California, Berkeley, fixated on the screen of a computer that is trying to decode someone's thoughts.
On the left-hand side of the screen is a reel of film clips that Gallant showed to a study participant during a brain scan. And on the right side of the screen, the computer program uses only the details of that scan to guess what the participant was watching at the time.
Anne Hathaway's face appears in a clip from the film Bride Wars, engaged in heated conversation with Kate Hudson. The algorithm confidently labels them with the words 'woman' and 'talk', in large type. Another clip appears — an underwater scene from a wildlife documentary. The program struggles, and eventually offers 'whale' and 'swim' in a small, tentative font.
“This is a manatee, but it doesn't know what that is,” says Gallant, talking about the program as one might a recalcitrant student. They had trained the program, he explains, by showing it patterns of brain activity elicited by a range of images and film clips. His program had encountered large aquatic mammals before, but never a manatee.
Groups around the world are using techniques like these to try to decode brain scans and decipher what people are seeing, hearing and feeling, as well as what they remember or even dream about.
Media reports have suggested that such techniques bring mind-reading “from the realms of fantasy to fact”, and “could influence the way we do just about everything”. The Economist in London even cautioned its readers to “be afraid”, and speculated on how long it will be until scientists promise telepathy through brain scans.
Nature Podcast: Decoding Brain Starts at 06:21
Although companies are starting to pursue brain decoding for a few applications, such as market research and lie detection, scientists are far more interested in using this process to learn about the brain itself. Gallant's group and others are trying to find out what underlies those different brain patterns and want to work out the codes and algorithms the brain uses to make sense of the world around it. They hope that these techniques can tell them about the basic principles governing brain organization and how it encodes memories, behavior and emotion (see 'Decoding for dummies').
Applying their techniques beyond the encoding of pictures and movies will require a vast leap in complexity. “I don't do vision because it's the most interesting part of the brain,” says Gallant. “I do it because it's the easiest part of the brain. It's the part of the brain I have a hope of solving before I'm dead.” But in theory, he says, “you can do basically anything with this”.
Brain decoding took off about a decade ago, when neuroscientists realized that there was a lot of untapped information in the brain scans they were producing using functional magnetic resonance imaging (fMRI). That technique measures brain activity by identifying areas that are being fed oxygenated blood, which light up as colored blobs in the scans. To analyze activity patterns, the brain is segmented into little boxes called voxels — the three-dimensional equivalent of pixels — and researchers typically look to see which voxels respond most strongly to a stimulus, such as seeing a face. By discarding data from the voxels that respond weakly, they conclude which areas are processing faces.