Times are tough for young psychologists. This thought has been rattling around in my head lately because we just finished searching for a new psychology professor at the university where I teach. When I met candidates, I had to ask about their field's troubling replication—and credibility—crisis.
I felt as though I was pressing them on some sordid personal matter, like whether alcoholism runs in their families, but the topic is unavoidable. A key test of the validity of any scientific study is whether its results can be replicated by other scientists. Last year a group called the Open Science Collaboration reported in Science that more than half of 100 studies published in major psychology journals had failed that test, despite painstaking efforts to re-create the original experiments.
The disappointing news was widely covered, including by Scientific American. In a front-page story, the New York Times declared that the report “confirmed the worst fears of scientists who have long worried that [psychology] needed a strong correction. The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory … the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.”
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
This past March the crisis made headlines again. A group of four prominent psychologists led by Daniel T. Gilbert of Harvard University claimed in Science that the 2015 Open Science Collaboration study was statistically flawed and did not prove its claim that “the reproducibility of psychological science is surprisingly low.” “Indeed,” Gilbert and his co-authors stated, “the data are consistent with the opposite conclusion, namely, that the reproducibility of psychological science is quite high.”
In a rebuttal, 44 authors involved in the Open Science Collaboration countered that the “very optimistic assessment” of Gilbert's group “is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data.” The exchange, Benedict Carey noted in the New York Times, “is likely to feed an already lively debate about how best to conduct and evaluate so-called replication projects of studies.”
That's too cheery an assessment. The exchange reveals that psychologists cannot even agree on basic methods for arriving at “truth,” whatever that is. As Katie M. Palmer pointed out in Wired, in an article headlined “Psychology Is in Crisis over Whether It's in Crisis,” “two groups of very smart people are looking at the exact same data and coming to wildly different conclusions.”
Meanwhile bad news keeps coming. A new study has raised doubts about the influential theory of “ego depletion,” which holds that willpower is a finite resource that diminishes with use.
In a 1998 paper that has now been cited thousands of times, Roy F. Baumeister and three other psychologists presented experimental evidence for ego depletion. The theory has supposedly been corroborated by hundreds of other studies, and it underpins the 2011 best seller Willpower: Rediscovering the Greatest Human Strength, by Baumeister and journalist John Tierney.
A multicenter team led by Martin Hagger and Nikos Chatzisarantis, both at Curtin University, recently tested the ego-depletion hypothesis in a study involving 2,141 subjects. In an unedited version of the paper released early by the Association for Psychological Science, the team concludes that “if there is any [ego-depletion] effect, it is close to zero.”
In a response, Baumeister and his colleague Kathleen D. Vohs dispute the methods of Hagger et al. but acknowledge that “this debacle shifts the burden of proof onto those of us who believe ego-depletion effects are genuine.” They plan to conduct their own replication study next year.
In a Slate article with the apocalyptic headline “Everything Is Crumbling,” journalist Daniel Engber notes that ego depletion is not “some crazy new idea, wobbling on a pile of flimsy data; it's a sturdy edifice of knowledge, built over many years from solid bricks…. If something this well established could fall apart, then what's next?” Good question, over which young psychologists are no doubt agonizing.
In the past, I've been hard on psychology, describing it as disturbingly faddish. Paradigms such as psychoanalysis and behaviorism never really die—they just go in and out of fashion. I like quoting linguist Noam Chomsky, who once said we will probably always learn more about ourselves from literature than from psychology. In that vein, I've argued that James Joyce's stream-of-consciousness masterpiece Ulysses gives us more insights into the working of our minds than any scientific study.
But perhaps because of my recent meetings with aspiring young psychologists, I'm feeling oddly protective toward the field. In fact, in an attempt to hearten psychologists young and old, I'd like to make the following four points:
First, there's nothing new about psychology's credibility crisis. More than a century ago William James worried that the field he helped to create might never transcend its “confused and imperfect state.” Howard Gardner argued in 1987 that “James's concerns have proved all too justified. Psychology has not added up to an integrated science, and it is unlikely ever to achieve that status.”
Second, psychologists are still doing important, empirically sound work. Two who recently spoke at my school are Sheldon Solomon, co-creator of terror-management theory, which predicts how fear of death affects us, and Philip Tetlock, leader of a study on “superforecasters,” ordinary people who do a better job than many so-called experts at predicting social phenomena.
Third, psychologists themselves have helped make us aware of how the quest for knowledge can go awry. Think of Daniel Kahneman's experiments on cognitive bias, on which he expounds in his blockbuster Thinking Fast and Slow, and Robert Trivers's research on self-deception, which he presents in The Folly of Fools. To help my students appreciate how we often see only what we're looking for, I show them the “invisible gorilla” video designed by Christopher Chabris and Daniel Simons.
Fourth, all scientific fields struggle with replication issues. Studies carried out over the past decade by statistician John Ioannidis have revealed that a large proportion of peer-reviewed claims turn out to be false. To my mind, behavioral genetics and psychiatry are much less credible than psychology, and string and multiverse theorists don't even have empirical results to replicate.
Psychology is arguably healthier than many other fields precisely because psychologists are energetically exposing its weaknesses and seeking ways to overcome them. I can't wait to chat about these issues next fall with my school's new psychology professor.
Editor's Note: This article was adapted from a post in the Cross-Check series.
