False positives and exaggerated results in peer-reviewed scientific studies have reached epidemic proportions in recent years. The problem is rampant in economics, the social sciences and even the natural sciences, but it is particularly egregious in biomedicine. Many studies that claim some drug or treatment is beneficial have turned out not to be true. We need only look to conflicting findings about beta-carotene, vitamin E, hormone treatments, Vioxx and Avandia. Even when effects are genuine, their true magnitude is often smaller than originally claimed.

The problem begins with the public’s rising expectations of science. Being human, scientists are tempted to show that they know more than they do. The number of investigators—and the number of experiments, observations and analyses they produce—has also increased exponentially in many fields, but adequate safeguards against bias are lacking. Research is fragmented, competition is fierce and emphasis is often given to single studies instead of the big picture.

Much research is conducted for reasons other than the pursuit of truth. Conflicts of interest abound, and they influence outcomes. In health care, research is often performed at the behest of companies that have a large financial stake in the results. Even for academics, success often hinges on publishing positive findings. The oligopoly of high-impact journals also has a distorting effect on funding, academic careers and market shares. Industry tailors research agendas to suit its needs, which also shapes academic priorities, journal revenue and even public funding.

The crisis should not shake confidence in the scientific method. The ability to prove something false continues to be a hallmark of science. But scientists need to improve the way they do their research and how they disseminate evidence.

First, we must routinely demand robust and extensive external validation—in the form of additional studies—for any report that claims to have found something new. Many fields pay little attention to the need for replication or do it sparingly and haphazardly. Second, scientific reports should take into account the number of analyses that have been conducted, which would tend to downplay false positives. Of course, that would mean some valid claims might get overlooked. Here is where large international collaborations may be indispensable. Human-genome epidemiology has recently had a good track record because several large-scale consortia rigorously validate genetic risk factors.

The best way to ensure that test results are verified would be for scientists to register their detailed experimental protocols before starting their research and disclose full results and data when the research is done. At the moment, results are often selectively reported, emphasizing the most exciting among them, and outsiders frequently do not have access to what they need to replicate studies. Journals and funding agencies should strongly encourage full public availability of all data and analytical methods for each published paper. It would help, too, if scientists stated up front the limitations of their data or inherent flaws in their study designs. Likewise, scientists and sponsors should be thorough in disclosing all potential conflicts of interest.

Some fields have adopted one or several of these mechanisms. Large international consortia are becoming commonplace in epidemiology; journals such as Annals of Internal Medicine and the Journal of the American Medical Association instruct authors to address study limitations; and many journals ask about conflicts of interest. Applying the measures widely won’t be easy, however.

Many scientists engaged in high-stakes research will refuse to make thorough disclosures. More important, much essential research has already been abandoned to the pharmaceutical and biomedical device industries, which may sometimes design and report studies in ways most favorable to their products. This is an embarrassment. Increased investment in evidence-based clinical and population research, for instance, should be designed not by industry but by scientists free of material conflicts of interest.

27 Comments

1. 1. HowardB 11:09 AM 5/31/11

   When an article starts out by blaming the public, it doesn't fill me with confidence in the analytical ability of the writer.
   The real cause, as the writer correctly then points out, is the fact that "Much research is conducted for reasons other than the pursuit of truth. Conflicts of interest abound, and they influence outcomes."
   Most research is either conducted by corporate sponsored science researchers or in the pursuit of continuing grants based on performance. The pressure to publish and publish real and positive results is higher than at any time in history.
   
   Some of the suggest 'solutions' offered are imho totally unworkable and undesirable. Emphasising "large international collaborations" would favour such large project work disproportionately and prejudice excellent and innovative work by small research groups in unexpected and new areas.
   
   The second real problem here is the almost complete breakdown in the peer review system. Papers with unfounded claims and unsubstantiated positive results are being passed routinely for publication. The bar has been lowered continuously over the last 25 years and the criteria for publication has been diluted.
   
   If a peer review system is to succeed and survive, then the chosen review peers should be supplied with all of the things suggested in this article even if they don't need to be included in the final published paper.. Evidence of replication, total numbers of analysis, details of experimental protocols, FULL data results, limitations and flaws. Failure to disclose should be a clear red flag fro publication.
   
   Rather than blaming the public, the blame and responsibility is squarely on the shoulders of the Scientific community. If it wants to raise standards then it must implement higher standards! it is that simple.
   
   Ultimately the peer review system as it stands is on it's last legs anyway. The future of published science has to be on the web, in a fully open and democratic forum. Critical review has to be opened up to the public and all other scientists, with full data disclosure, instead of the present system where only the well off can afford to buy scientific publications. Publishing was supposed to be the making available of Scientific work to the wider public ! Instead it has become a closed, narrow dissemination of Science to a tiny group of people.
   
   Quality work will stand up to criticism. Shoddy work will quickly flounder. The elitist attitude of the establishment toward such democratising of Science must be jettisoned.
   

   Reply | Report Abuse | Link to this

2. 2. davetrindle 11:35 AM 5/31/11

   Our science is very, very bad. One need only "look under the hood" of practically any published study to find flaws in methodology, flaws in the use of statistical measures, flaws in using adequately "controlled" study groups, and, of course, the MOTHER OF ALL INVALID SCIENTIFIC INFERENCES: mistaking correlation for cause-and-effect. We humans have far too elevated a view of our true intelligence and objectivity. In reality we are strongly subject to delusion, and "seeing what we want to see." Many of the greatest scientific "discoveries" have been refuted (e.g. that saturated fat causes heart disease). Many other "discoveries" have had a large element of luck, if not arrived at totally by accident.
   There should be some kind of minimum standard required before studies are published. It would reduce the publishing by 90%, and increase the validity of the studies by many multiples. The reduction in the constant flow of flawed studies would, in turn, allow more time for peer review of what actually gets published, which would further increase the quality.

   Reply | Report Abuse | Link to this

3. 3. davetrindle in reply to jimmywat 11:58 AM 5/31/11

   First, there is no evidence of "fraud," which would mean intentionally misleading studies. This was ruled out even in the case of the leaked emails, which simply showed the scientists were unprofessional. Their actual work was checked out and re-affirmed by independent, unbiased sources.
   Second, you CANNOT KNOW FOR SURE one way or the other about man-made global warming. You are speculating, based on incomplete information. Let's say there's a one-chance-in-a-million that current global warming theories are correct. Are you willing to make that bet? The downside is global human catastrophe beyond description, if not total extinction of the human race. Imagine if, for example, the growing season for 300 million Russians was substantially shortened or eliminated by a slight shift in weather patterns. Where will those people go? They certainly won't quietly starve to death. There would be World War beyond anything we have ever seen. And that is only one scenario among thousands.
   
   As we sort out the data, we should be seriously reducing our footprint before we destroy the human habitat. We should be studying the whole subject intensely. "Global warming isn't 'innocent until proven guilty' beyond the shadow of a doubt. Quite the opposite is. That's just common sense.
   (note: I agree with you on the biofuel fiasco.)

   Reply | Report Abuse | Link to this

4. 4. poosta7 12:04 PM 5/31/11

   The problem is that we have neither "evidence based medicine" or "evidence based belief systems". In either case, our beliefs underly the choices we make and our choices always have consequences. Ergo, we should try to base what we believe on the best evidence possible. The problem is that the "science" of belief creation is well understood by advertising and marketing firms. This results in societies that believe and make choices where there is no or even contradictory evidence. The other day my pastor told me that if I give money to the church (i.e. him) it makes God happy and when God is happy he will do nice things for me....plus the GM salesperson told me that if I buy a Corvette beautiful women will be attracted to me...and the toothpaste that gives me blazingly white teeth...so many things to believe and so many people to benefit from what I believe.

   Reply | Report Abuse | Link to this

5. 5. murraydunn 12:29 PM 5/31/11

   Thank you for speaking out about this.
   
   I for one would like to see Scientific American have more to say about the "scientific method" and the intellectual integrity required to do good science.

   Reply | Report Abuse | Link to this

6. 6. HowardB in reply to murraydunn 01:20 PM 5/31/11

   I agree.
   
   Unfortunately much as I love SciAm - it is very much a passive conduit for any and all Science reporting. As a reader of SciAm for decades it has always been clear that it is basically a financial and commercial venture who's goal is to be a conduit for Science research to the public. Hey there's nothing bad about being commercial, but it is important that this is kept in mind and a desire to hype research claims is an ever present tendency.
   
   I would love SciAm to lead the way in a campaign against bad science and for more scepticism and rational thinking. More 'critiquing' of the science that it reports is desperately needed. More robust insisting on adherence to the "Scientific method".
   
   I know it incites the believers but this includes more scepticism of the claims of the AGW establishment. At the heart of much of the AGW claims is a plethora of weak science such as the extrapolation of dendrochronology into the field of ancient climate analysis.
   
   Unfortunately challenging bad science is not necessarily a big money maker so I don't actually expect SciAm to change it's policies any time soon.

   Reply | Report Abuse | Link to this

7. 7. AngryNick in reply to murraydunn 01:54 PM 5/31/11

   I'd also like to see the educational system...say 7th and 8th grades...focus more on the _application_ of the scientific method and the importance of objectivity and integrity in science -- even when you don't like the results. While we may never be able to avoid honest mistakes or the fog of wishful thinking, our kids today seem to be suffering from a supreme lack of problem solving skills and the moral convictions required of real science.
   
   I recently had the opportunity to review the science projects of the top 7th grade science students in our area. To say it was disappointing is an understatement. Conclusions were weakly tied to the observations and results, but miraculously seemed to support the child's hypothesis. These same kids will be the doctors and researchers who will be curing my ailments in the a few years...and I'm not feeling so good about it.
   
   I think teaching kids to love the process of discovery is the key to making great scientist. Unfortunately, the focus in school is on getting an "A" at any cost.

   Reply | Report Abuse | Link to this

8. 8. murraydunn 02:40 PM 5/31/11

   I agree, but I believe the time to start science education is earlier.
   
   I've found many elementary school children have a strong desire to learn more about science, a desire that goes largely unmet at school. It seems such a waste that schools don't make a better effort at teaching such an important subject that their students are motivated to learn.
   
   While there are only so many hours in the classroom, and every hour is already filled with something, learning the scientific method is "strategically" important. Like learning to read, learning the scientific method opens up new avenues for children to make discoveries on their own. It gives them important tools to distinguish fact from fallacy in a world filled with individuals, industries and organization pedaling wu.
   
   Remember, that even if these children don't become scientists, one day they will be making decisions in their daily work, voting, and sitting on juries. Wouldn't you rather have them know what good science is.

   Reply | Report Abuse | Link to this

9. 9. blindboy 06:58 PM 5/31/11

   It would also help if reputable non-specialist science publications took a more skeptical attitude towards research. There have been numerous instances in recent years in which the substance of the research did not match the headline statement.
   This is barely forgiveable in mass media. In "scientific" publications it should simply not occur. "Some slight evidence that X influences Y" may not grab the attention of "X causes Y" but standards demand truth in headlining!

   Reply | Report Abuse | Link to this

10. 10. aluke33 10:18 PM 5/31/11

    This is why a Med School professor told my freshman class 50 some years ago: Remember, for every pearl on the ocean floor, there is one ton of whale manure. Way too much of the latter makes it into print.

    Reply | Report Abuse | Link to this

11. 11. Hel-n-highwater 12:14 PM 6/1/11

    First mention is the vitamin e controversy, aha boys as I look at my ankles I find that maybe some of you are really idiots. That Canadian doctor whose name I forget wrote an article and book that was published by Rodale in OGF, decades ago. I am glad that I started using Vit. E! Many people with varicose veins like I have get ulcerated legs and ankles. I have not. When I can not afforsd the vitamin e, the varicose veins increase and my ankles itch and are blue with the multiple venous arteries. I don't know who did the study but they don't convince me. People who listened to my narrative and used the vitamin e always watching not to overdose so that they had heart palipatations had my experience. It enables me to avoid all sorts of medical treatments. It costs medicare nothing but makes my life better. I know people who find that milk is a problem, some of our bodies are genetically susceptible to different things. My white heritage gave me MS but also the ability to absorb milk products. I have learned even though I am a college grad, not to believe every negative test for "old wives tales". Age does that for us if we keep an open mind.

    Reply | Report Abuse | Link to this

12. 12. GXF057 01:59 PM 6/1/11

    I agree with the comments by Howard that scientists need to take more responsibility for the review process. But reviewing papers is a sort of "community service" that we do in our spare time; so it's tough to ask people to do a better job of it. I don't have a good answer.
    
    I disagree with the suggestion that publishing should be open to anybody who wants to publish anything. Shoddy work does not necessarily flounder quickly. Take a look at "Merchants of Doubt" by Oreskes & Conway. The tobacco industry kept the 'scientific' discussion of the dangers of cigarette smoke going long after the evidence was established. Same with acid rain and ozone holes. If you have enough people who want to believe a proposition they'll find a way to promote their views. The idea of peer review is that *knowledgeable* people weigh in on the validity of the research before it gets published.
    -george
    

    Reply | Report Abuse | Link to this

13. 13. Timmy 08:15 AM 6/2/11

    What to do for researchers from developing countries who looks to publications in highly impacted journal as to gold standard of study quality. Quality of researches round the world is not equal and if gold standard is undermined that means that no need to reach best quality in trials. Or journal should openly say for all authors that in some cases they do not publish article not because of bad design but by reason if low interest by sponsor

    Reply | Report Abuse | Link to this

14. 14. jerrymat 03:09 PM 6/2/11

    As a high school science teacher (now retired) of more than a third of a century I think I know that the poor elementary teacher must prepare herself for so many different subjects that it is impossible to do a good job in all of them. The current emphasis is upon math and reading. I have talked to perfectly intellegent elementary teachers who routinely go through some route-learning sequence with their class as though science really operated that way. If the student can learn to spell "Hypothesis" he gets a good grade in science.
    
    Think about it. If a college elementary teacher prep. program finds that a student does well in science, where is the student directed? Into secondary science teaching - thereby depriving the elementary students of the best early science education they can get.
    
    In all of my years of high school teaching I met only one elementary teacher who actually taught hands on science to elementary students. I could always tell which of my freshmen came from his school.
    
    If we want to improve elementary science teaching then we need to fund summer institutes for elementary teachers so that they are paid to get the education they need to modify their instructional methods. I was able to go to such institutes for secondary science teachers in the 60s. Two of them gave me the knowledge to start a high school marine biology class. Such institutes for elementary teachers would do much to give children an early start in their science education.
    

    Reply | Report Abuse | Link to this

15. 15. Michael M 06:25 PM 6/2/11

    All readers miss the point that clinical studies, as a quite general rule, use far too small statistical samples - due to the ethical problem of experiment upon humans, and some other constraints.
    
    Clinical studies are like anecdote: undependable in themselves, but perhaps useful when a huge number of them can be included in metastudy base.
    
    Peer Review:
    A few journals have successfully strong peer review, and can be trusted. Most scientists know these.
    The go-to Med journals are fewer, but a couple are good.
    
    This issue is constantly addressed, and those who comment must make an effort to educate themselves on the ongoing observation and evaluation, BEFORE bewailing in generalities, science in general.
    
    The article is comprehensive, but it is you who must follow up with your own research, commentors, as the literature is far too extensive to be addressed in an article remotely as brief as it is necessary for the author to write. Above is journalism, and references would fill pages.
    
    Medicine is not science, do not confuse the two.
    
    Medicine uses various sciences, and although clinical studies can be statistically evaluated, leading untrained to believe that it is itself science, it has the constraints I mentioned, along with others, such as conflict of interest, which has skewed conclusions toward those which benefit the funders of both clinical and lab findings.
    
    There are far too many cases of fraud, exaggeration, and of failing to adjust for placebo effect. Due to our evolved human nature, many cases are not conscious fraud or exaggeration.
    
    To reiterate:
    Many many scientific journals have been created to serve corporate industrial interests.
    Many consumer health publications thrive due to acceptance of fraudulent and nonscientific claims made by producers who support overexuberant conclusions. These producers are incorporated in order to protect their investors/owners from personal liability. Government bureaus were intended to oversee such industries, as were groups such as AMA.
    Politics is the means by which "free market" confidence men and others who engage in fraud seek continued opportunity for personal gain.
    
    You must seek the kind of society in which you would prefer to live, knowing that our very nature devalues strangers, discounts the future for present gain, and rationalizes unethical behavior in large anonymous interaction.

    Reply | Report Abuse | Link to this

16. 16. Magmonster 07:32 PM 6/2/11

    Physician, heal thyself. SciAm has been wearing their politics on their sleeves and in their reader's faces for decades, which is why I canceled my subscription long ago. Politics doesn't belong in science any more than creationism in a science class. It's a form of intellectual corruption of the same type as is being discussed in this article - and a precursor of it as well. E.G. http://tinyurl.com/4y98zx6

    Reply | Report Abuse | Link to this

17. 17. rockyrcoon in reply to davetrindle 08:28 PM 6/2/11

    I agree that there is about 999,999 out of a million that man-made global warming isn't happening. For those odds, I would gladly not bet to hang an additional millstone around the neck of the US (and world) economy. That will cause way more innocent deaths than "global warming". And by the way, common sense would tell you that global warming would increase the growing season in Russia, if anything. Typical red herrings from the left....

    Reply | Report Abuse | Link to this

18. 18. Duster 07:05 PM 6/4/11

    It is correct more or less that there is no evidence of fraud per se in the AGW movement. What is far more likely is that the sincerity of proponents of AGW has interfered with their honesty, so to speak. They have taken the catastrophic outcomes of computer modeling seriously, assuming the adequacy of our knowledge of the climate system is adequate to model. Perforce it is reasonable to be alarmed to, explain their alarm, and to insist something be done. The problem is that AGW models as posed don't model or track reality.
    
    Assuming the physical properties of anthropic-generated CO2 are no different than those of natural CO2, there is no logical grounds to assume AGW computer model behaviour is any more than a mathematical artifact, and we should conclude that the models lacks any special empirical significance. Geological evidence indicates that CO2 has generally been many times higher in the past than it is at present - a near all-time low level. Thus, unless there is something inherently different about the physics of anthropic CO2, there is no creditable hazard posed by any foreseeable change in CO2 due to human activity. There many more serious anthropic threats to the environment than CO2 can ever be.

    Reply | Report Abuse | Link to this

19. 19. Kay42 10:47 PM 6/4/11

    Unbelievable. SciAm - what were you thinking?
    
    This is the identical problem "climate science" has. You have Nailed It.
    
    But you'll never admit to that, will you?

    Reply | Report Abuse | Link to this

20. 20. rwstutler in reply to Kay42 11:20 PM 6/4/11

    Kay 42 writes: "This is the identical problem "climate science" has."
    
    You are mistaken. The problem is that parody of science which is funded by for profit market actors - drug companies, oil/gas industries, etc. The parody they create looks like science and it is used as a marketing technique to government regulators and the buying public.
    
    Science funded by not for profit actors ("pure research") is less susceptible to becoming a parody of science and is more likely to be actual science at work.
    
    natural law, scientific methodology and practice - peer review, falsifiability and demonstrable fact tell us that human induced global warming is real.

    Reply | Report Abuse | Link to this

21. 21. Kay42 in reply to rwstutler 10:28 PM 6/5/11

    You are mistaken. Your talking points are old and tiresome. If drug companies and oil/gas industries are corrupt the entities that suck at the teat of government funding are equally corrupt.
    
    Grow up.

    Reply | Report Abuse | Link to this

22. 22. Kay42 10:38 PM 6/5/11

    Anyone who is still hanging on to this human-induced global warming foolishness is not doing their homework. A lively and satisfying discussion took place months ago at Climate Etc. See “Climate story telling angst” and all the following comments. Then come back here and spout your tripe.
    
    As Kip Hansen wrote at judithcurry.com:
    I must be shown the following:
    1) The hypothesis, which must be falsifiable.
    2) What experiment has been done and that it has been carefully laid out well enough to falsify your null hypothesis (and thus support your original hypothesis).
    3) What you did exactly, all the nasty details, how you controlled for every possible confounding factor (or didn’t control for this one and that…and why not, and how that might affect your findings).
    4) Your conclusion and how it follows from your data (and not your beliefs, feelings, hunches, or desire to please your funding agency or university tenure board).
    
    Then, and only then, will I listen to your opinion about what it might mean.
    

    Reply | Report Abuse | Link to this

23. 23. rwstutler in reply to Kay42 11:42 PM 6/5/11

    kay 42 writes: "If drug companies and oil/gas industries are corrupt the entities that suck at the teat of government funding are equally"
    
    sound, but invalid. thank you for considering the possibility that drug, oil and gas industries may be corrupt. you might also consider that the tobacco lobby/industry may be corrupt as well - meaning they have a financial interest in scientificaly proving, disproving or inciting controversy on certain issues. the monetary size of the research conducted by these potentially corrupt actors is significantly larger than the monetary size of the research conducted by not for profit (pure science) actors.

    Reply | Report Abuse | Link to this

24. 24. Kay42 09:00 AM 6/6/11

    Instead of reading what I wrote, you dragged in another tired example. Good job.
    
    Phil Jones brought in $22,000,000 in just one year. It's in a spreadsheet in the climate emails. The BBC has 33 billion dollars invested in the future of global warming. It would be a safe estimate to say that trillions of dollars have been "invested" in this scam.
    
    You are not interested in a discussion. You only believe in the evils of the free market system and the nirvana we would have if only the government controlled everything.
    
    Good luck with that.

    Reply | Report Abuse | Link to this

25. 25. rwstutler 04:30 PM 6/6/11

    kay 42 writes: "You only believe in the evils of the free market system and the nirvana we would have if only the government controlled everything."
    
    you are mistaken in your belief in the existence of a free market. and in your belief on my view of government.

    Reply | Report Abuse | Link to this

26. 26. bucketofsquid in reply to Magmonster 10:13 AM 6/8/11

    If you don't like the magazine and canceled your subscription, why are you still funding them by viewing their online articles and thus generating revenue via advertising? Put your money where your mouth is and stop financially supporting SciAm by never coming to their site. Those of us that like SciAm will gladly take your place.

    Reply | Report Abuse | Link to this

27. 27. trappedinamasonicworld in reply to Kay42 08:34 AM 2/16/12

    Quailty comment, hit's nail nicely on head...

    Reply | Report Abuse | Link to this