The researchers found seventy-four studies in total, representing 12,500 patients’ worth of data. Thirty-eight of these trials had positive results, and found that the new drug worked; thirty-six were negative. The results were therefore an even split between success and failure for the drugs, in reality. Then the researchers set about looking for these trials in the published academic literature, the material available to doctors and patients. This provided a very different picture. Thirty-seven of the positive trials—all but one—were published in full, often with much fanfare. But the trials with negative results had a very different fate: only three were published. Twenty-two were simply lost to history, never appearing anywhere other than in those dusty, disorganized, thin FDA files. The remaining eleven which had negative results in the FDA summaries did appear in the academic literature, but were written up as if the drug was a success. If you think this sounds absurd, I agree: we will see in Chapter 4, on ‘bad trials’, how a study’s results can be reworked and polished to distort and exaggerate its findings.
This was a remarkable piece of work, spread over twelve drugs from all the major manufacturers, with no stand-out bad guy. It very clearly exposed a broken system: in reality we have thirty-eight positive trials and thirty-seven negative ones; in the academic literature we have forty-eight positive trials and three negative ones. Take a moment to flip back and forth between those in your mind: “thirty-eight positive trials, thirty-seven negative”; or “forty-eight positive trials and only three negative”.
If we were talking about one single study, from one single group of researchers, who decided to delete half their results because they didn’t give the overall picture they wanted, then we would quite correctly call that act ‘research misconduct’. Yet somehow when exactly the same phenomenon occurs, but with whole studies going missing, by the hands of hundreds and thousands of individuals, spread around the world, in both the public and private sector, we accept it as a normal part of life. It passes by, under the watchful eyes of regulators and professional bodies who do nothing, as routine, despite the undeniable impact it has on patients.
Even more strange is this: we’ve known about the problem of negative studies going missing for almost as long as people have been doing serious science.
This was first formally documented by an American psychologist called Theodore Sterling in 1959. He went through every paper published in the four big psychology journals of the time, and found that 286 out of 294 reported a statistically significant result. This, he explained, was plainly fishy: it couldn’t possibly be a fair representation of every study that had been conducted, because if we believed that, we’d have to believe that almost every theory ever tested by a psychologist in an experiment had turned out to be correct. If psychologists really were so great at predicting results, there’d hardly be any point in bothering to run experiments at all. In 1995, at the end of his career, the same researcher came back to the same question, half a lifetime later, and found that almost nothing had changed.
Sterling was the first to put these ideas into a formal academic context, but the basic truth had been recognized for many centuries. Francis Bacon explained in 1620 that we often mislead ourselves by only remembering the times something worked, and forgetting those when it didn’t. Dr. Thomas Fowler in 1786 listed the cases he’d seen treated with arsenic, and pointed out that he could have glossed over the failures, as others might be tempted to do, but had included them. To do otherwise, he explained, would have been misleading.
Yet it was only three decades ago that people started to realize that missing trials posed a serious problem for medicine. In 1980 Elina Hemminki found that almost half the trials conducted in the mid-1970s in Finland and Sweden had been left unpublished. Then, in 1986, an American researcher called Robert Simes decided to investigate the trials on a new treatment for ovarian cancer. This was an important study, because it looked at a life-or-death question. Combination chemotherapy for this kind of cancer has very tough side effects, and knowing this, many researchers had hoped it might be better to give a single “alkylating agent” drug first, before moving on to full chemotherapy. Simes looked at all the trials published on this question in the academic literature, read by doctors and academics. From this, giving a single drug first looked like a great idea: women with advanced ovarian cancer (which is not a good diagnosis to have) who were on the alkylating agent alone were significantly more likely to survive longer.
See what we're tweeting about
18 Comments
Add CommentThis is an interesting commentary and it certainly has some merit in pointing out deficiencies in our current system of clinical trials. However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".
Reply | Report Abuse | Link to thisIt is likely that the trials sponsored by pharmaceutical companies are better funded, larger, and more carefully controlled. These trials are often admittedly designed to show efficacy - sometimes in a narrow population - as opposed to effectiveness. You simply have to do this to get a drug approved. There is nothing evil or inappropriate in this. If it doesn't work in a focused population, you might as well stop there. So, success and failure, independent versus pharma, need to be studied in similar trials otherwise you are comparing apples to oranges and committing the same sins of bias alluded to here. This is very difficult to do as pharma trials tend to be very different than independent trials in design, scope, and patient population. As to meta-analysis, namely, the analysis of a compilation of studies, this is extremely error prone and subject to bias and selection error as the author well knows.
Independently funded research is often conducted with an agenda no less biased than that of pharmaceutical companies. There are many people who set out with a goal, conscious or unconscious, of proving that a drug does not work and design their trials accordingly. For some, it seems good sport to be a thorn in the side of industry.
The truth is that nothing is perfect and perfect is the enemy of the good. There have been staggering R&D cuts in the pharmaceutical industry throughout the world due in large part from regulatory hurdles that demand clinical trials of unsupportable size, cost, and duration. This can chew up so much patent life and capital that companies simply give up.
Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research.
@deneb
Reply | Report Abuse | Link to this"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem""
It's clear he has the right kind of evidence to support his conclusions so I don't worry about tone and style.
Hi there,
Reply | Report Abuse | Link to thisI wrote the book that's extracted above.
"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem"."
There is a discussion at the end of each section, and the end of each chapter, and then at the end of the book, about what can be done to remedy these problems.
There is also a campaign at www.alltrials.net for the results of all trials on all currently used treatments to be reported in full, that has been signed by over 30,000 people, 80 patient groups, research funders, medical bodies, and the drug company GSK, who have signed up to our commitment:
http://www.badscience.net/2013/02/this-is-excellent-and-amazing-gsk-have-just-signed-up-to-alltrials-net/
Do please sign yourself:
www.alltrials.net
"Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research."
I wouldn't want that, nor do I ask for it. I simply ask that all results for all trials on all treatments used by patients are made available. It is impossible for doctors and patients to make informed decisions when results are withheld. Industry should be able to make money from good treatments (as they often do) without hiding information on them. I don't see that this is unreasonable, and you are also incorrect to assume that these issues only affect industry: academic studies are also frequently withheld, as I have discussed at length here and elsewhere.
Like many of the issues in the book, these issues are structural, and they are long overdue fixing.
at bengoldacre
Reply | Report Abuse | Link to thisThank you.
Most of normal us have this altruistic view, big pharma et al, like to hide everything so people don't get to see the real picture.
I signed your petition. :)
Shame we cant edit our posts to remove grammar errors though, as we race to type things.
Reply | Report Abuse | Link to this@Ben
Reply | Report Abuse | Link to thisThough I appreciate your campaign for openness, how are you planning to address the significant intellectual property rights aspect of funded research? It is a sticky issue, but don't you think that investors have the right to discover valuable information without giving it away to everyone, including their competitors?
Though patients and doctors need to make informed decisions, they are buying products from companies that have invested a great deal of money and skill into developing treatments that are often very easy and materially inexpensive to copy. Patents provide a limited amount of protection against the unfair competition that results when copycats have equal access to information.
I admit that I am troubled by the current situation and wish that information was more freely available. As taxpayers, we have the ability to ask our government to spend more of our money on publicly funded research, but our constitution is based on the principle that we do not have the right to take other people's property for public good without fair compensation.
Rod Adams
Publisher, Atomic Insights
Drug companies also undermine their own research by not disclosing what's in their placebos (and active placebos). They're allowed to manufacture their own placebos, some of which produce dry mouth and other symptoms (by design) and not report the ingredients. I have written about this recently in my blog at http://asserttrue.blogspot.com/2013/01/are-placebos-really-sugar-pills.html.
Reply | Report Abuse | Link to thisThank you for a great article.
Possible corrections to my comment:
Reply | Report Abuse | Link to this>However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".<
1) It's clear he has the right kind of evidence to support his conclusions so in this case I don't worry about tone or style.
2) I agree tone and style are important, but in this case they are so tame and the evidence is so strong there really is nothing to complain about.
3) I think the author does not show a 'clear bias against pharmaceutical research' but rather he shows 'clear and solid evidence against certain research practices in the pharmaceutical industry'.
Are you insane? Do you read your comments before posting them? When was the last time you checked your ethimometer?
Reply | Report Abuse | Link to thisI agree that all drug or therapy research has an agenda regardless of its funding source. However the gravy train of industry funded research is greased by mining for positive results regardless of how inane. While it is not fraud, it does create a lean to research literature that researchers, clinicians and others trained in the business recognize and defend against. This is one of the reasons pharmaceutical companies moved to direct consumer advertising. It is easier to create demand with images of happy families running through flower draped fields than by producing research papers that are going to be reviewed with a jaundiced eye by the prescriber.
Reply | Report Abuse | Link to thisAh, so! Those industry trials and surveys, are just as biased, as the ones, that I took, and often did not complete, because despite my answers, I was being led towards a conclusion, I could not support.
Reply | Report Abuse | Link to thisDitto political ones that tend to lead to a conclusion, the paid surveyor, promised his client.
No surprise here. The pharmaceutical industry (and chemical industry as a whole) have been cherry picking results of studies for decades. As far back as the early 1980s I wrote a series of stories about "cherry picking" that was so blatant that it was considered fraud in the testing of hundreds of food additives, drugs, pesticides, and industrial chemicals.
Reply | Report Abuse | Link to thisIndustrial Bio-Test Laboratories (IBT) was a particularly prolific source of toxicological testing. It also was involved in massive misconduct that resulted in the indictment and conviction of its president and a number of top executives in 1983. An audit by the FDA found that 71% of 867 studies were invalid because of "numerous discrepancies between the study conduct and data".
However, IBT was not an isolated case as the pharmaceutical and chemical industries claimed. Information I gathered from the FDA in 1980-1982 revealed that more than two dozen other labs had also engaged in significant levels of misconduct.
Three decades have passed and not much has changed when it comes to ensuring the integrity of safety studies on a massive number of products that we are exposed to.
As we move into an era in which biotechnology and nanotechnology will have an increased impact on our lives, the integrity of safety studies needs to be beyond question. Unfortunately, the lack of transparency makes it all to easy to continue to high unfavorable results.
AllTrials is a good start, but we need to ensure that not only results, but all of the data and study documentation are made available online so that many eyes can verify independently that "cherry picking", selective reporting, or outright fraud and fakery are not happening.
I'd say this has bias, I've seen an antidepressant drug research discontinuation seriously considered, don't konw what happened in the end, because more suicide attempts, although not reaching statistical signification, were in the experimental than in the comparator group. For pre-market trials, aimed to obtain a drug registration, this simply can't happen, Health Regulatory Authorities watch over all the process, and no fake data can be presented. Post-marketing studies are a different thing, as long as an study has not a Registered Trial Number, you cannot trust it, but Drug Surveillance data about unexpected side effects, not known in the early phases, when not too many patients had received the drug are even harder to hidden, as the economical consequences of hidding data are so enormous, that no company can seriously thinking it can be afforded, and a double registry, of drug producer and HRAs exists. There was another book on this more tan 20 years ago: "Dangerous medicine", I received it as a tip gift from a very nice person, but haven't found yet the time to read it, reality is often much more interesting than fiction.
Reply | Report Abuse | Link to thisA good read and well presented. Sadly, not very surprising findings given what I know about business and humans. You really can't help wanting to find positive results for the company you work for, where as an independent comes in with a skeptical eye.
Reply | Report Abuse | Link to thisThe details about the consequences of unpublished research are clear and important. It's unfortunate that the author starts with the statistics about how many studies support the drug being tested, because this is exactly the result you would expect if drug companies were doing what we want them to do in a highly ethical way.
Reply | Report Abuse | Link to thisLook, studies are expensive, so your "ideal" drug company is going to to do large, repeated studies on drugs they believe actually work. That is, they will re-test, in larger populations, the drugs that showed promising reults in small tirals. Unless drug companies are quite incompetent, the later trials will tend to confirm the ealier trials, producing exactly the result that the author presents as evidence of misconduct: most trials deonstrate the effectiveness of the drug being tested.
This lapse does not impugn the importance of the story he tells later about missing information in unpublished studies.
Clearly the past 'industry study' negative bias towards pharma companies has stopped with this article; Not. I agree that a case appears to have been made for greater transparency, but how would we know for sure with such blatant negative bias. Clearly there are countless more successful drugs and trials than there are questionable one. Lets not cost yet more lives by dramatic sensationalism and spooking the herd into a stampede of not taking drugs that do actual prolong and vastly improve people's lives.
Reply | Report Abuse | Link to thisI think a good start-up read to this book, one that I've already read and as a student of the pharmaceutical sciences, thoroughly enjoyed, was "Body Hunters" by Sonia Shah.
Reply | Report Abuse | Link to thisIt would seem to me that the high rate of positive results from pharma drug trials is simply because they are only willing to invest in a trial when they are already confident of a positive result.
Reply | Report Abuse | Link to this