Ed Vul

Ed Vul is a graduate student in the Department of Brain and Cognitive Sciences at the Massachusetts Intitute of Technology. He’s also the lead author of a recent paper, “Voodoo Correlations in Social Neuroscience,” which explored the high correlations between measures of personality or emotionality in the individual—such as the experience of fear, or the willingness to trust another person—with the activity of certain brain areas as observed in an fMRI machine. The paper has provoked a flurry of commentary. Mind Matters editor Jonah Lehrer chats with Vul about what this study means for the future of social neuroscience, whether the press is to blame and why we should always make multiple guesses.

LEHRER: What first got you interested in taking a critical look at fMRI papers in social neuroscience?

VUL: Some four years ago [University of California at San Diego neuroscientist] Hal Pashler and I saw a talk in which a very high correlation was reported between brain activity and the speed with which someone walked out of the room after the study.

Given what we knew about fMRI and the factors that determine how quickly we tend to walk in general, it seemed unbelievable to us that activity in this specific brain area could account for so much of the variance in walking speed. Especially so, because the fMRI activity was measured some two hours before the walking happened. So either activity in this area directly controlled motor action with a delay of two hours—something we found hard to believe—or there was something fishy going on. At that point, despite our suspicions, we didn't know exactly what that fishy thing was, so we put the topic aside.

A couple of years ago, I joined [M.I.T. neuroscientist] Nancy Kanwisher's lab and started working directly with fMRI data, and I learned the relevant jargon and statistics. At this point, [M.I.T. post-doc] Chris Baker and Nancy Kanwisher wrote a critique of a paper in Nature Neuroscience, which suffered from a non-independent analysis. After working through the general case myself (and writing a chapter on the topic), I realized how the correlations that seemed fishy to us so long ago were probably being produced, so we set out to investigate—ultimately leading to this paper.

LEHRER: What is a "voodoo correlation"?

VUL: We use that term as a humorous way to describe mysteriously high correlations produced by complicated statistical methods (which usually were never clearly described in the scientific papers we examined)—and which turn out unfortunately to yield some very misleading results. The specific issue we focus on, which is responsible for a great many mysterious correlations, is something we call “non-independent” testing and measurement of correlations. Basically, this involves inadvertently cherry-picking data and it results in inflated estimates of correlations.

To go into a bit more detail:

An fMRI scan produces lots of data: a 3-D picture of the head, which is divided into many little regions, called voxels. In a high-resolution fMRI scan, there will be hundreds of thousands of these voxels in the 3-D picture.

When researchers want to determine which parts of the brain are correlated with a certain aspect of behavior, they must somehow choose a subset of these thousands of voxels. One tempting strategy is to choose voxels that show a high correlation with this behavior. So far this strategy is fine.

The problem arises when researchers then go on to provide their readers with a quantative measure of the correlation magnitude measured just within the voxels they have pre-selected for having a high correlation. This two-step procedure is circular: it chooses voxels that have a high correlation, and then estimates a high average correlation. This practice inflates the correlation measurement because it selects those voxels that have benefited from chance, as well as any real underlying correlation, pushing up the numbers.