Image: Ed Vul
-
The Wisdom of Psychopaths
In this engrossing journey into the lives of psychopaths and their infamously crafty behaviors, the renowned psychologist Kevin Dutton reveals that there is a...
Read More »
Ed Vul is a graduate student in the Department of Brain and Cognitive Sciences at the Massachusetts Intitute of Technology. He’s also the lead author of a recent paper, “Voodoo Correlations in Social Neuroscience,” which explored the high correlations between measures of personality or emotionality in the individual—such as the experience of fear, or the willingness to trust another person—with the activity of certain brain areas as observed in an fMRI machine. The paper has provoked a flurry of commentary. Mind Matters editor Jonah Lehrer chats with Vul about what this study means for the future of social neuroscience, whether the press is to blame and why we should always make multiple guesses.
LEHRER: What first got you interested in taking a critical look at fMRI papers in social neuroscience?
VUL: Some four years ago [University of California at San Diego neuroscientist] Hal Pashler and I saw a talk in which a very high correlation was reported between brain activity and the speed with which someone walked out of the room after the study.
Given what we knew about fMRI and the factors that determine how quickly we tend to walk in general, it seemed unbelievable to us that activity in this specific brain area could account for so much of the variance in walking speed. Especially so, because the fMRI activity was measured some two hours before the walking happened. So either activity in this area directly controlled motor action with a delay of two hours—something we found hard to believe—or there was something fishy going on. At that point, despite our suspicions, we didn't know exactly what that fishy thing was, so we put the topic aside.
A couple of years ago, I joined [M.I.T. neuroscientist] Nancy Kanwisher's lab and started working directly with fMRI data, and I learned the relevant jargon and statistics. At this point, [M.I.T. post-doc] Chris Baker and Nancy Kanwisher wrote a critique of a paper in Nature Neuroscience, which suffered from a non-independent analysis. After working through the general case myself (and writing a chapter on the topic), I realized how the correlations that seemed fishy to us so long ago were probably being produced, so we set out to investigate—ultimately leading to this paper.
LEHRER: What is a "voodoo correlation"?
VUL: We use that term as a humorous way to describe mysteriously high correlations produced by complicated statistical methods (which usually were never clearly described in the scientific papers we examined)—and which turn out unfortunately to yield some very misleading results. The specific issue we focus on, which is responsible for a great many mysterious correlations, is something we call “non-independent” testing and measurement of correlations. Basically, this involves inadvertently cherry-picking data and it results in inflated estimates of correlations.
To go into a bit more detail:
An fMRI scan produces lots of data: a 3-D picture of the head, which is divided into many little regions, called voxels. In a high-resolution fMRI scan, there will be hundreds of thousands of these voxels in the 3-D picture.
When researchers want to determine which parts of the brain are correlated with a certain aspect of behavior, they must somehow choose a subset of these thousands of voxels. One tempting strategy is to choose voxels that show a high correlation with this behavior. So far this strategy is fine.
The problem arises when researchers then go on to provide their readers with a quantative measure of the correlation magnitude measured just within the voxels they have pre-selected for having a high correlation. This two-step procedure is circular: it chooses voxels that have a high correlation, and then estimates a high average correlation. This practice inflates the correlation measurement because it selects those voxels that have benefited from chance, as well as any real underlying correlation, pushing up the numbers.
See what we're tweeting about
10 Comments
Add CommentGreat discussion and a great paper. It's been interesting and somewhat disturbing to see social neuroscience taken-up with so much gusto in questions of law and responsibility. Authors of these studies and commentaries often make sweeping conclusions suggesting that we dispense with traditional notions of justice because "brain scans show x". It's unfortunate that more scholars haven't had a healthy dose of skepticism when these claims were (and are) made. Perhaps this paper can bring some balance to the discussion.
Reply | Report Abuse | Link to thisI totally agree, I think to push science further to new and exciting paradigms, we need to question everything that is shown to us, and then some. Keep up the great work and keep asking questions!
Reply | Report Abuse | Link to thisyes...seems some of these claims are little more than the modern analogue of phrenology.
Reply | Report Abuse | Link to thisIt really isn't media that is hyping this pseudoscience as much as it is the universities and their own pr machines. See for instance the 2008 news release from the University of Wisconsin, Madison: STUDY SHOWS COMPASSION MEDITATION CHANGES THE BRAIN hyping Richard Davidson's human experimentation.
Reply | Report Abuse | Link to thisI just wanted to point out the very recent reply to Vue's paper, by Lieberman et al (2009, in press). Sorry but seems that "Voodoo correlations..." were just another example of mindless press hype over crap study.
Reply | Report Abuse | Link to thisRegards :-)
I just wanted to point out the very recent reply to Vue's paper, by Lieberman et al (2009, in press). Sorry but seems that "Voodoo correlations..." were just another example of mindless press hype over crap study.
Reply | Report Abuse | Link to thisRegards :-)
Sorry for posting twice. Hopefully you check the paper I mentioned, and be a bit more sceptical about what you read in press :)
Reply | Report Abuse | Link to thisEdward Val is still a graduate student, not even in the field of neuroimaging, and certainly not qualified to make the claims that he tried to make in this paper. His "study" has been resoundingly struck down by the true experts in the relevant fields, who collectively have hundreds more years of experience. I wouldn't be surprised if his paper is retracted before it ever reaches print.
Reply | Report Abuse | Link to thisMy personal opinion is that he showed great immaturity and poor science in trying to come up with such overreaching sensational press-friendly claims.
The last few comments are disturbing, as they seem to attack a person, rather then a point. But, they are also misleading. Especially, the response of John in LA that Vul's "study has been resoundingly struck down by the true experts in the relevant fields". Serious readers should read to the actual Vul article in the journal PPS and the commentaries where everyone, except certain expert from LA, agrees with the essence of Vuls criticism. You can also go to any recent neuro-blog (Neurocritic, Neurosceptic) and form their own opinion.
Reply | Report Abuse | Link to thisSorry David, but have you already checked the paper I mentioned? I've already uploaded it on scribd (shouldn't be also difficult to find it via Google). This one seems to be a convincing rebuttal of Vul's et al. study. Where do you see ad personam?
Reply | Report Abuse | Link to this