Image: Illustration by Christian Northeast
-
What a Plant Knows
How does a Venus flytrap know when to snap shut? Can it actually feel an insect’s tiny, spindly legs? And how do cherry blossoms know when to bloom? Can they...
Read More »
As of this writing, the precise nature of Marc Hauser’s transgressions remains murky. Hauser is Harvard’s superstar primate psychologist—and, perhaps ironically, an expert on the evolution of morality—whom the university recently found guilty of eight counts of scientific misconduct. Harvard has kept mum about the details, but a former lab assistant alleged that when Hauser looked at videotapes of rhesus monkeys, in an experiment on their capacity to learn sound patterns, he noted behavior that other people in the lab couldn’t see, in a way that consistently favored his hypothesis. When confronted with these discrepancies, the assistant says, Hauser asserted imperiously that his interpretation was right and the others’ wrong.
Hauser has admitted to committing “significant mistakes.” In observing the reactions of my colleagues to Hauser’s shocking comeuppance, I have been surprised at how many assume reflexively that his misbehavior must have been deliberate. For example, University of Maryland physicist Robert L. Park wrote in a Web column that Hauser “fudged his experiments.” I don’t think we can be so sure. It’s entirely possible that Hauser was swayed by “confirmation bias”—the tendency to look for and perceive evidence consistent with our hypotheses and to deny, dismiss or distort evidence that is not.
The past few decades of research in cognitive, social and clinical psychology suggest that confirmation bias may be far more common than most of us realize. Even the best and the brightest scientists can be swayed by it, especially when they are deeply invested in their own hypotheses and the data are ambiguous. A baseball manager doesn’t argue with the umpire when the call is clear-cut—only when it is close.
Scholars in the behavioral sciences, including psychology and animal behavior, may be especially prone to bias. They often make close calls about data that are open to many interpretations. Last year, for instance, Belgian neurologist Steven Laureys insisted that a comatose man could communicate through a keyboard, even after controlled tests failed to find evidence. Climate researchers trying to surmise past temperature patterns by using proxy data are also engaged in a “particularly challenging exercise because the data are incredibly messy,” says David J. Hand, a statistician at Imperial College London.
Two factors make combating confirmation bias an uphill battle. For one, data show that eminent scientists tend to be more arrogant and confident than other scientists. As a consequence, they may be especially vulnerable to confirmation bias and to wrong-headed conclusions, unless they are perpetually vigilant. Second, the mounting pressure on scholars to conduct single-hypothesis-driven research programs supported by huge federal grants is a recipe for trouble. Many scientists are highly motivated to disregard or selectively reinterpret negative results that could doom their careers. Yet when members of the scientific community see themselves as invulnerable to error, they impede progress and damage the reputation of science in the public eye. The very edifice of science hinges on the willingness of investigators to entertain the possibility that they might be wrong.
The best antidote to fooling ourselves is adhering closely to scientific methods. Indeed, history teaches us that science is not a monolithic truth-gathering method but rather a motley assortment of tools designed to safeguard us against bias. In the behavioral sciences, such procedures as control groups, blinded designs and independent coding of data are essential methodological bulwarks against bias. They minimize the odds that our hypotheses will mislead us into seeing things that are not there and blind us from seeing things that are. As astronomer Carl Sagan and his wife and co-author Ann Druyan noted, science is like a little voice in our heads that says, “You might be mistaken. You’ve been wrong before.” Good scientists are not immune from confirmation bias. They are aware of it and avail themselves of procedural safeguards against its pernicious effects.
This article was originally published with the title Fudge Factor.
Already a Digital subscriber? Sign-in Now
If your institution has site license access, enter here.
Rights & Permissions
See what we're tweeting about
13 Comments
Add CommentThis applies as well to the confirmation bias exhibited by the IPCC, Michael Mann, Phil Jones, Bradley, Hughes, Briffa and all the rest of the government and university climate scientists, who apparently believe in their computer models – even though those models have an abysmal record and are consistently wrong.
Reply | Report Abuse | Link to thisThose same scientists do not seem interested in getting out into the mud and the slush and the weeds, and doing actual testable, physical experiments and verifiable observations. That would induce a bad case of Leon Festinger's cognitive dissonance. It is much easier to believe their favorite AGW hypothesis.
In my applied experience, the process of producing an accurate model include both verification (prediction of actual results based on actual conditions) and validation (prediction of future results based on expected conditions). Validation in particular requires the publication of predicted results in advance of actual conditions which must confirm the veracity of predictions. This process must include prior publication of results to prevent later modifications to the model (fudge factors) used to make the model match actual results. I don't see this procedure followed much, but I wouldn't bet my future on any model that hasn't been properly confirmed by formal validation.
Reply | Report Abuse | Link to thisNice column, especially the final paragraph, However, I'd like to remind you (and Michael Shermer) that modern times have no lock on good science. Almost four hundred years ago, in 1620, Francis Bacon published Novum Organum(1). In that he describes the four idols of the mind. These are the classifications of the prejudices that interfere with rational thought. While Bacon rambles on (and on), making Novum Organum difficult to read, he was spot on when it came to establishing the scientific method. Briefly, the four idols are the Idols of the Tribe, characteristic of all humans. These include supposing a greater order and uniformity in things than can be shown. Also included are dismissing evidence that disagrees with a hypothesis, and giving too much weight to that which gives support, etc. There are also the Idols of the Cave (e.g., pride that distorts research), the Market (e.g., uncritical acceptance of commonalities), and the Theater (misleading uncritical thinkers). Modern experts, such as the redoubtable Carl Sagan, may provide an umbra of authority. However, Scientific American columnists should be aware that four hundred years ago Francis Bacon said it all quite well, thank you.
Reply | Report Abuse | Link to this(1) Urbach & Gibson, transl. and ed., "Francis Bacon, Novum Organum," (Chicago, 1994).
I would hope that a columnist for SA would have done their research, but apparently not.
Reply | Report Abuse | Link to thisThe evidence against Hauser is far beyond "confirmation bias". The author just mentions one of the eight counts of misconduct that Hauser has been found guilty of. He also seems, for example, to have fabricated data in a paper he had to retract from Cognition. According to the Cognition editor (who has inside knowledge of the Harvard investigation), the control data in the Cognition paper was *fabricated*; the control data are not in the videos and thus do not exist. The NY Times claimed that the Cognition editor (Altmann) says that this could have been accidental, but Altmann says in his blog that the NY Times misrepresented him and that there is virtually no other explanation than fabrication.
The author of this column also ignores the fact that Harvard found Hauser guilty of misconduct and Harvard's definition of misconduct does not include innocent mistakes like confirmation bias.
Those of us in the Harvard community have been assured by the Dean that Hauser is guilty, his guilt is clear, and it was deliberate misconduct, including deliberate falsification and fabrication. There have been rumors about Hauser fabricating data for years. It was common knowledge among his students that they had to protect their data from him or he would change the results. Unconscious confirmation bias? Ridiculous!
A column about confirmation bias is welcome. It makes an important point. But this has nothing to do with Hauser. Once the NIH issues their report, this will be clear to everyone, including columnists who don't do their homework.
@Macuser,
Reply | Report Abuse | Link to this"This applies as well to the confirmation bias exhibited by the IPCC, Michael Mann, Phil Jones, Bradley, Hughes, Briffa and all the rest of the government and university climate scientists, who apparently believe in their computer models even though those models have an abysmal record and are consistently wrong."
No, that is just you reciting unthinkingly fossil fuel propaganda. That you are unaware that Dr. Mann's results have replicated a dozen times by different researchers, using different methods and proxies is only par for the course.
@macuser
Reply | Report Abuse | Link to thisConflating the work of Hauser with the climate scientists you reference is what is wrong. Hauser's work has apparently been found wanting in the scientific community. The work of IPCC et. al. has been attacked in the media and blogosphere based on illegally obtained e-mails. The e-mails suggest juvenile behavior in the face of harassment, but claims that the e-mails are evidence of deceit misrepresent what they actually say.
I would be interested in any data (not unsubstantiated claims) that the models have an abysmal record and would welcome any model that has a better record.
Your argument that the climate modeling is somehow deficient is specious as well, suggesting by analogy that the designer or builder of a racing car must also drive the thing as well. There are a number of scientists validating the predictions of climate models as we see actual evidence of rising CO2 levels, melting ice caps, mass wasting of glaciers, the signature of the greenhouse effect in satellite data and the warmest decade on record punctuated by extreme weather events.
In my limited exposure to research in the fields of cognitive, social and clinical psychology, and branches of sociology in general appear to me to also be highly negligent in conducting statistical analyses of, especially, subjects that are selected primarily on their meeting cost and convenience criteria, being students at the researchers' educational institutions. Similar cost restrictions seem to apply in compromising the validity of representative sampling in popular fMRI studies of neuroscientists.
Reply | Report Abuse | Link to thisIt seems that most research reports I read in these pages extrapolate even social characteristics of limited university student populations to the entire population of humanity, generalizing fast food studies to dietary nutrition preferences, for example.
While these biases occur not in the analysis phase but in the selection phase of testing, they can have an equally significant influence in the quality of results obtained. Efforts to minimize research costs should not be allowed to significantly influence methods and results. Research conducted improperly is better left undone, regardless of any publication quotas that may apply.
I was going to make the same points. It's annoying when a columnist cherry picks aspects of a case like this as an excuse to write the column they want.
Reply | Report Abuse | Link to thisI agree with Frank - you can't pick and choose what you want & call it 'evidence' to prove your point.
Reply | Report Abuse | Link to thisOf late, many like to accuse the science community of grandstanding. It appears to the public that we make many grand claims that in the end mostly fall short of their claimed promises. Such is the necessity of obtaining funding.
Reply | Report Abuse | Link to thisWhile it is true the frequency of unsavory conduct in the research world appears to be on the increase it is important to bear one thing in mind: at least the science community has a system in place to discover and act against such transgressions. It may not be as clean as we want or as fast but at least we have one. Try to find a similar system that actually works in any other area of our culture.
Most business activities are operated on a relatively strict Return On Investment basis. Business projects that falsify their ROI expectations eventually produce inefficient businesses that typically fail in the marketplace.
Reply | Report Abuse | Link to thisResearchers can apparently obtain funding without such strict accountability. For example, I give you the Large Hadron Collider, which will apparently require an enormous amount of additional funding despite the severe economic condition of several of its funding states.
Granted, most publicly funded projects are not held strictly accountable for their expenditures...
Acting, speaking and thinking as a Social Scientist, I've often wondered how we dare to call ourselves "Scientists?" Can anyone deny that there is more interpretative observation, non-objective extrapolation, convergent basic assumptions, and skewed intuition then application of The Scientific Method?
Reply | Report Abuse | Link to thisIts impossible to deny that, in our fields of interest, the elements of Art are most surely present. And its all because the objects/subjects of our interests are human beings rather than atomic weight tables or mathematical calcutions. To avoid being held to the same standards as an astrophysicist or biochemist, maybe we ought to change the name of our broad field of endeavor to the Social Arts?
Hello Friends, I read this forum i like it I m agree with u.Thanks
Reply | Report Abuse | Link to this----
<a href="http://www.casualdate.net.au/melbourne-swingers" rel="dofollow">Melbourne swingers</a>