This article was published in Scientific American’s former blog network and reflects the views of the author, not necessarily those of Scientific American
It’s a tough time to be a young psychologist. This thought keeps occurring to me as we search for a new psychology professor at my school, Stevens Institute of Technology. When I meet candidates, I have to ask about their field’s replication—and credibility—crisis.
I feel as though I’m pressing them on some sordid personal matter, like whether alcoholism runs in their families, but the topic is unavoidable. Last summer, a group called the “Open Science Collaboration” reported in Science that it had replicated fewer than half of 100 studies published in major psychology journals.
The New York Times declared in a front-page story that the report “confirmed the worst fears of scientists who have long worried that [psychology] needed a strong correction. The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions, and the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.”
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
The crisis keeps generating headlines. On Friday, a group of four prominent psychologists led by Daniel Gilbert of Harvard claimed in Science that last year’s Open Collaboration study was statistically flawed and did not prove its claim that “the reproducibility of psychological science is low.” “Indeed,” Gilbert and his co-authors state, “the data are consistent with the opposite conclusion, namely, that the reproducibility of psychological science is quite high.”
In a rebuttal, 44 authors involved in the Open Science Collaboration countered that the “very optimistic assessment” of Gilbert’s group “is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data.”
The exchange, Benedict Carey notes in the Times, “is likely to feed an already lively debate about how best to conduct and evaluate so-called replication projects of studies.” That’s too cheery an assessment. The exchange reveals that psychologists cannot even agree on basic methods for arriving at “truth,” whatever that is.
It gets worse. Over at Slate, Daniel Engber reports that the influential theory of “ego depletion”—which holds that willpower is a finite resource that diminishes with use--might have been “debunked.”
Roy Baumeister and three other psychologists presented experimental evidence for the theory in a 1998 paper that has been cited more than 3,000 times. The theory has supposedly been corroborated by hundreds of other studies, and it underpins the 2011 bestseller Willpower: Rediscovering the Greatest Human Strength, by Baumeister and journalist John Tierney.
But a study of ego depletion involving “more than 2,000 subjects tested at two-dozen different labs on several continents,” Engber reports, found “exactly nothing. A zero-effect for ego depletion: No sign that the human will works as it’s been described, or that these hundreds of studies amount to very much at all.” The new study is scheduled for publication next month in Perspectives on Psychological Science.
Spelling out the disturbing implications, Engber notes that the ego-depletion effect “has been recreated in hundreds of different ways, and the underlying concept has been verified via meta-analysis. It’s not some crazy new idea, wobbling on a pile of flimsy data; it’s a sturdy edifice of knowledge, built over many years from solid bricks. And yet, it now appears that ego depletion could be completely bogus, that its foundation might be made of rotted-out materials. That means an entire field of study—and significant portions of certain scientists’ careers—could be resting on a false premise. If something this well-established could fall apart, then what’s next?”
Good question, over which all young psychologists are no doubt agonizing. To cheer themselves, they might consider the following four points:
First, there’s nothing new about psychology’s credibility crisis. More than a century ago, William James worried that the field he helped create might never transcend its “confused and imperfect state.”
Second, all scientific fields struggle with replication issues. Behavioral genetics and psychiatry are arguably much less credible than psychology, and string and multiverse theorists don’t even have empirical results to replicate!
Third, psychologists are still doing important, empirically sound work. Two who recently spoke at my school are Sheldon Solomon, co-creator of terror-management theory, which predicts how fear of death affects us; and Philip Tetlock, leader of a study on “superforecasters,” ordinary people who do a better job than many so-called experts at predicting social phenomena.
Fourth, psychology is arguably healthier than many other fields precisely because psychologists are energetically exposing its weaknesses and seeking ways to overcome them.
I look forward to discussing these issues with the young psychologists visiting my school.
Further Reading:
Meta-Post: Horgan Posts on Brain and Mind Science.
A Dig Through Old Files Reminds Me Why I’m So Critical of Science.
