In the rough-and-tumble world of science, disputes are usually settled in time, as a convergence of evidence accumulates in favor of one hypothesis over another. Until now.
On April 10 economist John R. Lott, Jr., formerly of the American Enterprise Institute, filed a defamation lawsuit against economist Steven D. Levitt of the University of Chicago and HarperCollins, the publisher of Levitt's 2005 book, Freakonomics. At issue is what Levitt meant when he wrote that scholars could not "replicate" Lott's results, referring to Lott's 1998 book, More Guns, Less Crime. Lott employed a sophisticated statistical analysis on data from state-level variation in "carry and conceal" laws, finding that states that passed laws permitting citizens to carry concealed weapons saw statistically significant declines in robbery, rape and homicide compared with states that did not pass such laws.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Replicating results means testing hypotheses.
As is typical with such politically charged research, considerable controversy followed publication of Lott's book, with a flurry of conference presentations and journal papers, some of which replicated his results and some of which did not. For example, in a series of papers published in the Stanford Law Review (available at http://papers.ssrn.com), Lott and his critics debated the evidence.
In Freakonomics, Levitt proffered his own theory for the source of the 1990s crime decline--Roe v. Wade. According to Levitt, children born into impoverished and adverse environments are more likely to land in jail as adults. After Roe v. Wade, millions of poor single women had abortions instead of future potential criminals; 20 years later the set of potential offenders had shrunk, along with the crime rate. Levitt employed a comparative statistical analysis to show that the five states that legalized abortion at least two years before Roe v. Wade witnessed a crime decline earlier than the other 45 states. Further, those states with the highest abortion rates in the 1970s experienced the greatest fall in crime in the 1990s.
One factor that Levitt dismissed is Lott's, in a single -passage in the middle of a 30-page chapter: "Lott's admittedly intriguing hypothesis doesn't seem to be true. When other scholars have tried to replicate his results, they found that right-to-carry laws simply don't bring down crime."
According to Lott's legal complaint, "the term 'replicate' has an objective and factual meaning": that other scholars "have analyzed the identical data that Lott analyzed and analyzed it the way Lott did in order to determine whether they can reach the same result." When Levitt said that they could not, he was "alleging that Lott falsified his results."
I asked Levitt what he meant by "replicate." He replied: "I used the term in the same way that most scientists do--substantiate results." Substantiate, not duplicate. Did he mean to imply that Lott falsified his results? "No, I did not." In fact, others have accused Lott of falsifying his data, so I asked Lott why he is suing Levitt. "Having some virtually unheard-of people making allegations on the Internet is one thing," Lott declared. "Having claims made in a book published by an economics professor and printed by a reputable book publisher, already with sales exceeding a million copies, is something entirely different. In addition, Levitt is well known, and his claims unfortunately carry some weight. I have had numerous people ask me after reading Freakonomics whether it is really true that others have been unable to replicate my research."
"Replicate" is a verb that depends on the sentence's object. "Replicate methodology" might capture Lott's meaning, but "replicate results" means testing the conclusion of the methodology, in this case that having more guns in society results in less crime. The problem is that such analyses are so complicated that the failure to replicate more likely indicates modeling mistakes made during the original research or in the replication process rather than fakery. [break]
Mr. Lott, tear down this legal wall and let us return to doing science without lawyers. Replicating results means testing hypotheses--not merely duplicating methodologies--and this central tenet of science can only flourish in an atmosphere of open peer review.