One can see closely analogous phenomena in many areas of life. Suppose we pick out the investment analysts whose stock picks for April 2005 did best for that month. These people will probably tend to have talent going for them, but they will also have had unusual luck (and some finance experts, such as Nassim Taleb, actually say the luck will probably be the bigger element). But even assuming they are more talented than average—as we suspect they would be—if we ask them to predict again, for some later month, we will invariably find that as a group, they cannot duplicate the performance they showed in April. The reason is that next time, luck will help some of them and hurt some of them—whereas in April, they all had luck on their side or they wouldn’t have gotten into the top group. So their average performance in April is an overestimate of their true ability—the performance they can be expected to duplicate on the average month.
It is exactly the same with fMRI data and voxels. If researchers select only highly correlated voxels, they select voxels that "got lucky," as well as having some underlying correlation. So if you take the correlations you used to pick out the voxels as a measure of the true correlation for these voxels, you will get a very misleading overestimate.
This, then, is what we think is at the root of the voodoo correlations: the analysis inadvertently capitalized on chance, resulting in inflated measurements of correlation. The tricky part, which I can’t go into here, was that investigators were actually trying to take account of the fact they were checking so many different brain areas—but their precautions made the problem that I am describing worse, not better!
LEHRER: Your paper has prompted a great deal of debate among social neuroscientists, and some of the scientists have issued a rebuttal of your paper. (You have since rebutted this rebuttal.) What do you hope this debate leads to? What methodological changes would you like to see adopted by social neuroscientists using fMRI?
VUL: The debate we have spurred is quite interesting to watch. At first some of the authors whose papers we criticized challenged our statistical point, but—for good reason--that line of argument doesn’t seem to have caught on. Right now, so far as I know, everyone seems to concede that the analysis used in these studies was not kosher, in the sense of providing correlation numbers that can be taken seriously. Instead, we are mostly hearing a couple of other arguments at this point.
One is that the correlation values themselves don’t really matter—it’s just the fact there is a correlation in a certain spot in the head that matters. I don’t agree with this observation at all, and we think the fact that many of these papers appeared in such high profile places is because editors were (justifiably) impressed with big effects. If one can account for, say, three quarters of individual differences in something important such as anxiety or empathy—obviously, that’s a real breakthrough, and it tells you not only where future research ought to look, but also where it shouldn’t. On the other hand, if it’s just 3 percent of the variance, that’s a whole lot less impressive, and may reflect much more indirect kinds of associations.
I have also heard some people complain that even if we are right on the mathematical point, we presented our argument in a bit of a rough-mannered way—criticizing particular articles and drawing unfavorable outside attention to the field, and using the humorous term “voodoo.”
We were as surprised as anyone by how much interest our paper sparked. Evidently it spread sort of “virally”—one neuroscientist we know said he got seven copies sent to him (none of them by us). The good side is that people are thinking harder now about how they do their analyses. The bad side is that all this publicity has left some authors feeling embarrassed and picked on. In our view, the statistical issues of independence and multiple comparisons are full of tricky pitfalls—we do not suggest that these were stupid mistakes people were making, and we regret hurting anyone’s feelings. I don’t think, however, that it would have made sense to write an article that did not “name names,” because if the scientific literature is to guide future research decisions, people have to know which results can be relied upon, and which cannot. (In fact, we suspect we only flagged a small fraction of the papers that have these problems, and some are in other fields, such as neurogenetics, cognitive neuroscience more broadly, and others.)
See what we're tweeting about
10 Comments
Add CommentGreat discussion and a great paper. It's been interesting and somewhat disturbing to see social neuroscience taken-up with so much gusto in questions of law and responsibility. Authors of these studies and commentaries often make sweeping conclusions suggesting that we dispense with traditional notions of justice because "brain scans show x". It's unfortunate that more scholars haven't had a healthy dose of skepticism when these claims were (and are) made. Perhaps this paper can bring some balance to the discussion.
Reply | Report Abuse | Link to thisI totally agree, I think to push science further to new and exciting paradigms, we need to question everything that is shown to us, and then some. Keep up the great work and keep asking questions!
Reply | Report Abuse | Link to thisyes...seems some of these claims are little more than the modern analogue of phrenology.
Reply | Report Abuse | Link to thisIt really isn't media that is hyping this pseudoscience as much as it is the universities and their own pr machines. See for instance the 2008 news release from the University of Wisconsin, Madison: STUDY SHOWS COMPASSION MEDITATION CHANGES THE BRAIN hyping Richard Davidson's human experimentation.
Reply | Report Abuse | Link to thisI just wanted to point out the very recent reply to Vue's paper, by Lieberman et al (2009, in press). Sorry but seems that "Voodoo correlations..." were just another example of mindless press hype over crap study.
Reply | Report Abuse | Link to thisRegards :-)
I just wanted to point out the very recent reply to Vue's paper, by Lieberman et al (2009, in press). Sorry but seems that "Voodoo correlations..." were just another example of mindless press hype over crap study.
Reply | Report Abuse | Link to thisRegards :-)
Sorry for posting twice. Hopefully you check the paper I mentioned, and be a bit more sceptical about what you read in press :)
Reply | Report Abuse | Link to thisEdward Val is still a graduate student, not even in the field of neuroimaging, and certainly not qualified to make the claims that he tried to make in this paper. His "study" has been resoundingly struck down by the true experts in the relevant fields, who collectively have hundreds more years of experience. I wouldn't be surprised if his paper is retracted before it ever reaches print.
Reply | Report Abuse | Link to thisMy personal opinion is that he showed great immaturity and poor science in trying to come up with such overreaching sensational press-friendly claims.
The last few comments are disturbing, as they seem to attack a person, rather then a point. But, they are also misleading. Especially, the response of John in LA that Vul's "study has been resoundingly struck down by the true experts in the relevant fields". Serious readers should read to the actual Vul article in the journal PPS and the commentaries where everyone, except certain expert from LA, agrees with the essence of Vuls criticism. You can also go to any recent neuro-blog (Neurocritic, Neurosceptic) and form their own opinion.
Reply | Report Abuse | Link to thisSorry David, but have you already checked the paper I mentioned? I've already uploaded it on scribd (shouldn't be also difficult to find it via Google). This one seems to be a convincing rebuttal of Vul's et al. study. Where do you see ad personam?
Reply | Report Abuse | Link to this