I am setting out this evidence at length because I want to be absolutely clear that there is no doubt on the issue. Industry sponsored trials give favorable results, and that is not my opinion, or a hunch from the occasional passing study. This is a very well-documented problem, and it has been researched extensively, without anybody stepping out to take effective action, as we shall see.
There is one last study I’d like to tell you about. It turns out that this pattern of industry-funded trials being vastly more likely to give positive results persists even when you move away from published academic papers, and look instead at trial reports from academic conferences, where data often appears for the first time (in fact, as we shall see, sometimes trial results only appear at an academic conference, with very little information on how the study was conducted).
Fries and Krishnan studied all the research abstracts presented at the 2001 American College of Rheumatology meetings which reported any kind of trial, and acknowledged industry sponsorship, in order to find out what proportion had results that favored the sponsor’s drug. There is a small punch-line coming, and to understand it we need to cover a little of what an academic paper looks like. In general, the results section is extensive: the raw numbers are given for each outcome, and for each possible causal factor, but not just as raw figures. The ‘ranges’ are given, subgroups are perhaps explored, statistical tests are conducted, and each detail of the result is described in table form, and in shorter narrative form in the text, explaining the most important results. This lengthy process is usually spread over several pages.
In Fries and Krishnan [2004] this level of detail was unnecessary. The results section is a single, simple, and—I like to imagine—fairly passive-aggressive sentence:
The results from every RCT (45 out of 45) favored the drug of the sponsor.
This extreme finding has a very interesting side effect, for those interested in time-saving shortcuts. Since every industry sponsored trial had a positive result, that’s all you’d need to know about a piece of work to predict its outcome: if it was funded by industry, you could know with absolute certainty that the trial found the drug was great.
How does this happen? How do industry-sponsored trials almost always manage to get a positive result? It is, as far as anyone can be certain, a combination of factors. Sometimes trials are flawed by design. You can compare your new drug with something you know to be rubbish—an existing drug at an inadequate dose, perhaps, or a placebo sugar pill that does almost nothing. You can choose your patients very carefully, so they are more likely to get better on your treatment. You can peek at the results halfway through, and stop your trial early if they look good (which is—for interesting reasons we shall discuss—statistical poison). And so on.
But before we get to these fascinating methodological twists and quirks, these nudges and bumps that stop a trial from being a fair test of whether a treatment works or not, there is something very much simpler at hand.
Sometimes drug companies conduct lots of trials, and when they see that the results are unflattering, they simply fail to publish them. This is not a new problem, and it’s not limited to medicine. In fact, this issue of negative results that go missing in action cuts into almost every corner of science. It distorts findings in fields as diverse as brain imaging and economics, it makes a mockery of all our efforts to exclude bias from our studies, and despite everything that regulators, drug companies and even some academics will tell you, it is a problem that has been left unfixed for decades.
In fact, it is so deep-rooted that even if we fixed it today—right now, for good, forever, without any flaws or loopholes in our legislation—that still wouldn’t help, because we would still be practicing medicine, cheerfully making decisions about which treatment is best, on the basis of decades of medical evidence which is—as you’ve now seen—fundamentally distorted.
See what we're tweeting about
18 Comments
Add CommentThis is an interesting commentary and it certainly has some merit in pointing out deficiencies in our current system of clinical trials. However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".
Reply | Report Abuse | Link to thisIt is likely that the trials sponsored by pharmaceutical companies are better funded, larger, and more carefully controlled. These trials are often admittedly designed to show efficacy - sometimes in a narrow population - as opposed to effectiveness. You simply have to do this to get a drug approved. There is nothing evil or inappropriate in this. If it doesn't work in a focused population, you might as well stop there. So, success and failure, independent versus pharma, need to be studied in similar trials otherwise you are comparing apples to oranges and committing the same sins of bias alluded to here. This is very difficult to do as pharma trials tend to be very different than independent trials in design, scope, and patient population. As to meta-analysis, namely, the analysis of a compilation of studies, this is extremely error prone and subject to bias and selection error as the author well knows.
Independently funded research is often conducted with an agenda no less biased than that of pharmaceutical companies. There are many people who set out with a goal, conscious or unconscious, of proving that a drug does not work and design their trials accordingly. For some, it seems good sport to be a thorn in the side of industry.
The truth is that nothing is perfect and perfect is the enemy of the good. There have been staggering R&D cuts in the pharmaceutical industry throughout the world due in large part from regulatory hurdles that demand clinical trials of unsupportable size, cost, and duration. This can chew up so much patent life and capital that companies simply give up.
Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research.
@deneb
Reply | Report Abuse | Link to this"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem""
It's clear he has the right kind of evidence to support his conclusions so I don't worry about tone and style.
Hi there,
Reply | Report Abuse | Link to thisI wrote the book that's extracted above.
"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem"."
There is a discussion at the end of each section, and the end of each chapter, and then at the end of the book, about what can be done to remedy these problems.
There is also a campaign at www.alltrials.net for the results of all trials on all currently used treatments to be reported in full, that has been signed by over 30,000 people, 80 patient groups, research funders, medical bodies, and the drug company GSK, who have signed up to our commitment:
http://www.badscience.net/2013/02/this-is-excellent-and-amazing-gsk-have-just-signed-up-to-alltrials-net/
Do please sign yourself:
www.alltrials.net
"Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research."
I wouldn't want that, nor do I ask for it. I simply ask that all results for all trials on all treatments used by patients are made available. It is impossible for doctors and patients to make informed decisions when results are withheld. Industry should be able to make money from good treatments (as they often do) without hiding information on them. I don't see that this is unreasonable, and you are also incorrect to assume that these issues only affect industry: academic studies are also frequently withheld, as I have discussed at length here and elsewhere.
Like many of the issues in the book, these issues are structural, and they are long overdue fixing.
at bengoldacre
Reply | Report Abuse | Link to thisThank you.
Most of normal us have this altruistic view, big pharma et al, like to hide everything so people don't get to see the real picture.
I signed your petition. :)
Shame we cant edit our posts to remove grammar errors though, as we race to type things.
Reply | Report Abuse | Link to this@Ben
Reply | Report Abuse | Link to thisThough I appreciate your campaign for openness, how are you planning to address the significant intellectual property rights aspect of funded research? It is a sticky issue, but don't you think that investors have the right to discover valuable information without giving it away to everyone, including their competitors?
Though patients and doctors need to make informed decisions, they are buying products from companies that have invested a great deal of money and skill into developing treatments that are often very easy and materially inexpensive to copy. Patents provide a limited amount of protection against the unfair competition that results when copycats have equal access to information.
I admit that I am troubled by the current situation and wish that information was more freely available. As taxpayers, we have the ability to ask our government to spend more of our money on publicly funded research, but our constitution is based on the principle that we do not have the right to take other people's property for public good without fair compensation.
Rod Adams
Publisher, Atomic Insights
Drug companies also undermine their own research by not disclosing what's in their placebos (and active placebos). They're allowed to manufacture their own placebos, some of which produce dry mouth and other symptoms (by design) and not report the ingredients. I have written about this recently in my blog at http://asserttrue.blogspot.com/2013/01/are-placebos-really-sugar-pills.html.
Reply | Report Abuse | Link to thisThank you for a great article.
Possible corrections to my comment:
Reply | Report Abuse | Link to this>However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".<
1) It's clear he has the right kind of evidence to support his conclusions so in this case I don't worry about tone or style.
2) I agree tone and style are important, but in this case they are so tame and the evidence is so strong there really is nothing to complain about.
3) I think the author does not show a 'clear bias against pharmaceutical research' but rather he shows 'clear and solid evidence against certain research practices in the pharmaceutical industry'.
Are you insane? Do you read your comments before posting them? When was the last time you checked your ethimometer?
Reply | Report Abuse | Link to thisI agree that all drug or therapy research has an agenda regardless of its funding source. However the gravy train of industry funded research is greased by mining for positive results regardless of how inane. While it is not fraud, it does create a lean to research literature that researchers, clinicians and others trained in the business recognize and defend against. This is one of the reasons pharmaceutical companies moved to direct consumer advertising. It is easier to create demand with images of happy families running through flower draped fields than by producing research papers that are going to be reviewed with a jaundiced eye by the prescriber.
Reply | Report Abuse | Link to thisAh, so! Those industry trials and surveys, are just as biased, as the ones, that I took, and often did not complete, because despite my answers, I was being led towards a conclusion, I could not support.
Reply | Report Abuse | Link to thisDitto political ones that tend to lead to a conclusion, the paid surveyor, promised his client.
No surprise here. The pharmaceutical industry (and chemical industry as a whole) have been cherry picking results of studies for decades. As far back as the early 1980s I wrote a series of stories about "cherry picking" that was so blatant that it was considered fraud in the testing of hundreds of food additives, drugs, pesticides, and industrial chemicals.
Reply | Report Abuse | Link to thisIndustrial Bio-Test Laboratories (IBT) was a particularly prolific source of toxicological testing. It also was involved in massive misconduct that resulted in the indictment and conviction of its president and a number of top executives in 1983. An audit by the FDA found that 71% of 867 studies were invalid because of "numerous discrepancies between the study conduct and data".
However, IBT was not an isolated case as the pharmaceutical and chemical industries claimed. Information I gathered from the FDA in 1980-1982 revealed that more than two dozen other labs had also engaged in significant levels of misconduct.
Three decades have passed and not much has changed when it comes to ensuring the integrity of safety studies on a massive number of products that we are exposed to.
As we move into an era in which biotechnology and nanotechnology will have an increased impact on our lives, the integrity of safety studies needs to be beyond question. Unfortunately, the lack of transparency makes it all to easy to continue to high unfavorable results.
AllTrials is a good start, but we need to ensure that not only results, but all of the data and study documentation are made available online so that many eyes can verify independently that "cherry picking", selective reporting, or outright fraud and fakery are not happening.
I'd say this has bias, I've seen an antidepressant drug research discontinuation seriously considered, don't konw what happened in the end, because more suicide attempts, although not reaching statistical signification, were in the experimental than in the comparator group. For pre-market trials, aimed to obtain a drug registration, this simply can't happen, Health Regulatory Authorities watch over all the process, and no fake data can be presented. Post-marketing studies are a different thing, as long as an study has not a Registered Trial Number, you cannot trust it, but Drug Surveillance data about unexpected side effects, not known in the early phases, when not too many patients had received the drug are even harder to hidden, as the economical consequences of hidding data are so enormous, that no company can seriously thinking it can be afforded, and a double registry, of drug producer and HRAs exists. There was another book on this more tan 20 years ago: "Dangerous medicine", I received it as a tip gift from a very nice person, but haven't found yet the time to read it, reality is often much more interesting than fiction.
Reply | Report Abuse | Link to thisA good read and well presented. Sadly, not very surprising findings given what I know about business and humans. You really can't help wanting to find positive results for the company you work for, where as an independent comes in with a skeptical eye.
Reply | Report Abuse | Link to thisThe details about the consequences of unpublished research are clear and important. It's unfortunate that the author starts with the statistics about how many studies support the drug being tested, because this is exactly the result you would expect if drug companies were doing what we want them to do in a highly ethical way.
Reply | Report Abuse | Link to thisLook, studies are expensive, so your "ideal" drug company is going to to do large, repeated studies on drugs they believe actually work. That is, they will re-test, in larger populations, the drugs that showed promising reults in small tirals. Unless drug companies are quite incompetent, the later trials will tend to confirm the ealier trials, producing exactly the result that the author presents as evidence of misconduct: most trials deonstrate the effectiveness of the drug being tested.
This lapse does not impugn the importance of the story he tells later about missing information in unpublished studies.
Clearly the past 'industry study' negative bias towards pharma companies has stopped with this article; Not. I agree that a case appears to have been made for greater transparency, but how would we know for sure with such blatant negative bias. Clearly there are countless more successful drugs and trials than there are questionable one. Lets not cost yet more lives by dramatic sensationalism and spooking the herd into a stampede of not taking drugs that do actual prolong and vastly improve people's lives.
Reply | Report Abuse | Link to thisI think a good start-up read to this book, one that I've already read and as a student of the pharmaceutical sciences, thoroughly enjoyed, was "Body Hunters" by Sonia Shah.
Reply | Report Abuse | Link to thisIt would seem to me that the high rate of positive results from pharma drug trials is simply because they are only willing to invest in a trial when they are already confident of a positive result.
Reply | Report Abuse | Link to this