How Neuroscientists Observe Brains Watching Movies

Functional MRI can peer inside your brain and watch you watching a YouTube clip

Next, they applied the same Bayesian framework to decode fMRI signals. They used 5,000 hours’ worth of short clips pulled at random from YouTube to establish a baseline of “natural movies.” The same three subjects were tested by watching movies in the magnet they had not previously seen and that were not drawn from the natural movies data set. The decoder estimates the most likely clip based on the response of many voxels in the visual cortex of each volunteer. It is a very sophisticated form of hedging one’s bets based on prior experience, widely used in a variety of applications—such as predicting that your credit card is being misused by somebody who has very different purchasing patterns.

Reconstructing the movie in the head leads to some stunning results. (I urge the reader to visit Gallant’s Web site, where a movie highlights the side-by-side comparison between viewed and decoded movies.) The method is far from perfect—the reconstructed clips are slow and lack details. After all, the fMRI signal is read out only once every second, whereas the underlying movies are much more dynamic (with a 15-hertz frame rate). Yet the net result is astounding, even for an old hand like me.

What Does the Future Hold?
As our measurement tools become more precise and our algorithms more sophisticated, the quality of the reconstructed movies will improve. Indeed, it is not inconceivable that the kind of visual daydreaming we all engage in—sexual fantasies, the crux of the climb where I keep on falling, what I should have told my boss—will one day yield to these tools (provided that I engage in imagery while lying completely immobile in a magnetic scanner). And who’s to say that dreams might not also be accessible to Gallant’s reconstruction techniques?

Functional brain imaging is perfectly safe and requires nothing more than reclining on one’s back uncomfortably for a few hours in a tight metal cylinder. Yet the fundamental spatiotemporal limits of fMRI remain. It does not access the atoms of perception, individual neurons. At the moment, only intrusive microelectrodes that are implanted in the brains of some patients, as was described in my May/June 2011 column, can access the substrate out of which our most fleeting experiences, thoughts and conscious memories arise. For now these remain safe from prying eyes.

(Further Reading)

  • Bayesian Reconstruction of Natural Images from Human Brain Activity. Thomas Naselaris, Ryan J. Prenger, Kendrick N. Kay, Michael Oliver and Jack L. Gallant in Neuron, Vol. 63, No. 6, pages 902–915; September 24, 2009.
  • Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies. Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu and Jack L. Gallant in Current Biology, published online September 22, 2011.

This article was originally published with the title "Consciousness Redux: Movies in the Cortical Theater."

or subscribe to access other articles from the January 2012 publication.
Digital Issue $7.95
Digital Subscription $19.99 Subscribe
Share this Article:


You must sign in or register as a member to submit a comment.

Give a Gift &
Get a Gift - Free!

Give a 1 year subscription
as low as $9.99

Subscribe Now! >


Email this Article