In a few pages, we will walk through the literature that demonstrates all of this beyond any doubt, showing that “publication bias”—the process whereby negative results go unpublished—is endemic throughout the whole of medicine and academia; and that regulators have failed to do anything about it, despite decades of data showing the size of the problem. But before we get to that research, I need you to feel its implications, so we need to think about why missing data matters.
Evidence is the only way we can possibly know if something works—or doesn’t work—in medicine. We proceed by testing things, as cautiously as we can, in head-to-head trials, and gathering together all of the evidence. This last step is crucial: if I withhold half the data from you, it’s very easy for me to convince you of something that isn’t true. If I toss a coin a hundred times, for example, but only tell you about the results when it lands heads-up, I can convince you that this is a two-headed coin. But that doesn’t mean I really do have a two-headed coin: it means I’m misleading you, and you’re a fool for letting me get away with it. This is exactly the situation we tolerate in medicine, and always have. Researchers are free to do as many trials as they wish, and then choose which ones to publish.
The repercussions of this go way beyond simply misleading doctors about the benefits and harms of interventions for patients, and way beyond trials. Medical research isn’t an abstract academic pursuit: it’s about people, so every time we fail to publish a piece of research we expose real, living people to unnecessary, avoidable suffering.
TGN1412
In March 2006, six volunteers arrived at a London hospital to take place in a trial. It was the first time a new drug called TGN1412 had ever been given to humans, and they were paid £2,000 each. Within an hour these six men developed headaches, muscle aches, and a feeling of unease. Then things got worse: high temperatures, restlessness, periods of forgetting who and where they were. Soon they were shivering, flushed, their pulses racing, their blood pressure falling. Then, a cliff: one went into respiratory failure, the oxygen levels in his blood falling rapidly as his lungs filled with fluid. Nobody knew why. Another dropped his blood pressure to just 65/40, stopped breathing properly, and was rushed to an intensive care unit, knocked out, intubated, mechanically ventilated. Within a day all six were disastrously unwell: fluid on their lungs, struggling to breathe, their kidneys failing, their blood clotting uncontrollably throughout their bodies, and their white blood cells disappearing. Doctors threw everything they could at them: steroids, antihistamines, immune-system receptor blockers. All six were ventilated on intensive care. They stopped producing urine; they were all put on dialysis; their blood was replaced, first slowly, then rapidly; they needed plasma, red cells, platelets. The fevers continued. One developed pneumonia. And then the blood stopped getting to their peripheries. Their fingers and toes went flushed, then brown, then black, and then began to rot and die. With heroic effort, all escaped, at least, with their lives.
The Department of Health convened an Expert Scientific Group to try to understand what had happened, and from this two concerns were raised. First: can we stop things like this from happening again? It’s plainly foolish, for example, to give a new experimental treatment to all six participants in a ‘first-in-man’ trial at the same time, if that treatment is a completely unknown quantity. New drugs should be given to participants in a staggered process, slowly, over a day. This idea received considerable attention from regulators and the media.
Less noted was a second concern: could we have foreseen this disaster? TGN1412 is a molecule that attaches to a receptor called CD28 on the white blood cells of the immune system. It was a new and experimental treatment, and it interfered with the immune system in ways that are poorly understood, and hard to model in animals (unlike, say, blood pressure, because immune systems are very variable between different species). But as the final report found, there was experience with a similar intervention: it had simply not been published. One researcher presented the inquiry with unpublished data on a study he had conducted in a single human subject a full ten years earlier, using an antibody that attached to the CD3, CD2 and CD28 receptors. The effects of this antibody had parallels with those of TGN1412, and the subject on whom it was tested had become unwell. But nobody could possibly have known that, because these results were never shared with the scientific community. They sat unpublished, unknown, when they could have helped save six men from a terrifying, destructive, avoidable ordeal.
See what we're tweeting about
18 Comments
Add CommentThis is an interesting commentary and it certainly has some merit in pointing out deficiencies in our current system of clinical trials. However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".
Reply | Report Abuse | Link to thisIt is likely that the trials sponsored by pharmaceutical companies are better funded, larger, and more carefully controlled. These trials are often admittedly designed to show efficacy - sometimes in a narrow population - as opposed to effectiveness. You simply have to do this to get a drug approved. There is nothing evil or inappropriate in this. If it doesn't work in a focused population, you might as well stop there. So, success and failure, independent versus pharma, need to be studied in similar trials otherwise you are comparing apples to oranges and committing the same sins of bias alluded to here. This is very difficult to do as pharma trials tend to be very different than independent trials in design, scope, and patient population. As to meta-analysis, namely, the analysis of a compilation of studies, this is extremely error prone and subject to bias and selection error as the author well knows.
Independently funded research is often conducted with an agenda no less biased than that of pharmaceutical companies. There are many people who set out with a goal, conscious or unconscious, of proving that a drug does not work and design their trials accordingly. For some, it seems good sport to be a thorn in the side of industry.
The truth is that nothing is perfect and perfect is the enemy of the good. There have been staggering R&D cuts in the pharmaceutical industry throughout the world due in large part from regulatory hurdles that demand clinical trials of unsupportable size, cost, and duration. This can chew up so much patent life and capital that companies simply give up.
Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research.
@deneb
Reply | Report Abuse | Link to this"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem""
It's clear he has the right kind of evidence to support his conclusions so I don't worry about tone and style.
Hi there,
Reply | Report Abuse | Link to thisI wrote the book that's extracted above.
"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem"."
There is a discussion at the end of each section, and the end of each chapter, and then at the end of the book, about what can be done to remedy these problems.
There is also a campaign at www.alltrials.net for the results of all trials on all currently used treatments to be reported in full, that has been signed by over 30,000 people, 80 patient groups, research funders, medical bodies, and the drug company GSK, who have signed up to our commitment:
http://www.badscience.net/2013/02/this-is-excellent-and-amazing-gsk-have-just-signed-up-to-alltrials-net/
Do please sign yourself:
www.alltrials.net
"Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research."
I wouldn't want that, nor do I ask for it. I simply ask that all results for all trials on all treatments used by patients are made available. It is impossible for doctors and patients to make informed decisions when results are withheld. Industry should be able to make money from good treatments (as they often do) without hiding information on them. I don't see that this is unreasonable, and you are also incorrect to assume that these issues only affect industry: academic studies are also frequently withheld, as I have discussed at length here and elsewhere.
Like many of the issues in the book, these issues are structural, and they are long overdue fixing.
at bengoldacre
Reply | Report Abuse | Link to thisThank you.
Most of normal us have this altruistic view, big pharma et al, like to hide everything so people don't get to see the real picture.
I signed your petition. :)
Shame we cant edit our posts to remove grammar errors though, as we race to type things.
Reply | Report Abuse | Link to this@Ben
Reply | Report Abuse | Link to thisThough I appreciate your campaign for openness, how are you planning to address the significant intellectual property rights aspect of funded research? It is a sticky issue, but don't you think that investors have the right to discover valuable information without giving it away to everyone, including their competitors?
Though patients and doctors need to make informed decisions, they are buying products from companies that have invested a great deal of money and skill into developing treatments that are often very easy and materially inexpensive to copy. Patents provide a limited amount of protection against the unfair competition that results when copycats have equal access to information.
I admit that I am troubled by the current situation and wish that information was more freely available. As taxpayers, we have the ability to ask our government to spend more of our money on publicly funded research, but our constitution is based on the principle that we do not have the right to take other people's property for public good without fair compensation.
Rod Adams
Publisher, Atomic Insights
Drug companies also undermine their own research by not disclosing what's in their placebos (and active placebos). They're allowed to manufacture their own placebos, some of which produce dry mouth and other symptoms (by design) and not report the ingredients. I have written about this recently in my blog at http://asserttrue.blogspot.com/2013/01/are-placebos-really-sugar-pills.html.
Reply | Report Abuse | Link to thisThank you for a great article.
Possible corrections to my comment:
Reply | Report Abuse | Link to this>However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".<
1) It's clear he has the right kind of evidence to support his conclusions so in this case I don't worry about tone or style.
2) I agree tone and style are important, but in this case they are so tame and the evidence is so strong there really is nothing to complain about.
3) I think the author does not show a 'clear bias against pharmaceutical research' but rather he shows 'clear and solid evidence against certain research practices in the pharmaceutical industry'.
Are you insane? Do you read your comments before posting them? When was the last time you checked your ethimometer?
Reply | Report Abuse | Link to thisI agree that all drug or therapy research has an agenda regardless of its funding source. However the gravy train of industry funded research is greased by mining for positive results regardless of how inane. While it is not fraud, it does create a lean to research literature that researchers, clinicians and others trained in the business recognize and defend against. This is one of the reasons pharmaceutical companies moved to direct consumer advertising. It is easier to create demand with images of happy families running through flower draped fields than by producing research papers that are going to be reviewed with a jaundiced eye by the prescriber.
Reply | Report Abuse | Link to thisAh, so! Those industry trials and surveys, are just as biased, as the ones, that I took, and often did not complete, because despite my answers, I was being led towards a conclusion, I could not support.
Reply | Report Abuse | Link to thisDitto political ones that tend to lead to a conclusion, the paid surveyor, promised his client.
No surprise here. The pharmaceutical industry (and chemical industry as a whole) have been cherry picking results of studies for decades. As far back as the early 1980s I wrote a series of stories about "cherry picking" that was so blatant that it was considered fraud in the testing of hundreds of food additives, drugs, pesticides, and industrial chemicals.
Reply | Report Abuse | Link to thisIndustrial Bio-Test Laboratories (IBT) was a particularly prolific source of toxicological testing. It also was involved in massive misconduct that resulted in the indictment and conviction of its president and a number of top executives in 1983. An audit by the FDA found that 71% of 867 studies were invalid because of "numerous discrepancies between the study conduct and data".
However, IBT was not an isolated case as the pharmaceutical and chemical industries claimed. Information I gathered from the FDA in 1980-1982 revealed that more than two dozen other labs had also engaged in significant levels of misconduct.
Three decades have passed and not much has changed when it comes to ensuring the integrity of safety studies on a massive number of products that we are exposed to.
As we move into an era in which biotechnology and nanotechnology will have an increased impact on our lives, the integrity of safety studies needs to be beyond question. Unfortunately, the lack of transparency makes it all to easy to continue to high unfavorable results.
AllTrials is a good start, but we need to ensure that not only results, but all of the data and study documentation are made available online so that many eyes can verify independently that "cherry picking", selective reporting, or outright fraud and fakery are not happening.
I'd say this has bias, I've seen an antidepressant drug research discontinuation seriously considered, don't konw what happened in the end, because more suicide attempts, although not reaching statistical signification, were in the experimental than in the comparator group. For pre-market trials, aimed to obtain a drug registration, this simply can't happen, Health Regulatory Authorities watch over all the process, and no fake data can be presented. Post-marketing studies are a different thing, as long as an study has not a Registered Trial Number, you cannot trust it, but Drug Surveillance data about unexpected side effects, not known in the early phases, when not too many patients had received the drug are even harder to hidden, as the economical consequences of hidding data are so enormous, that no company can seriously thinking it can be afforded, and a double registry, of drug producer and HRAs exists. There was another book on this more tan 20 years ago: "Dangerous medicine", I received it as a tip gift from a very nice person, but haven't found yet the time to read it, reality is often much more interesting than fiction.
Reply | Report Abuse | Link to thisA good read and well presented. Sadly, not very surprising findings given what I know about business and humans. You really can't help wanting to find positive results for the company you work for, where as an independent comes in with a skeptical eye.
Reply | Report Abuse | Link to thisThe details about the consequences of unpublished research are clear and important. It's unfortunate that the author starts with the statistics about how many studies support the drug being tested, because this is exactly the result you would expect if drug companies were doing what we want them to do in a highly ethical way.
Reply | Report Abuse | Link to thisLook, studies are expensive, so your "ideal" drug company is going to to do large, repeated studies on drugs they believe actually work. That is, they will re-test, in larger populations, the drugs that showed promising reults in small tirals. Unless drug companies are quite incompetent, the later trials will tend to confirm the ealier trials, producing exactly the result that the author presents as evidence of misconduct: most trials deonstrate the effectiveness of the drug being tested.
This lapse does not impugn the importance of the story he tells later about missing information in unpublished studies.
Clearly the past 'industry study' negative bias towards pharma companies has stopped with this article; Not. I agree that a case appears to have been made for greater transparency, but how would we know for sure with such blatant negative bias. Clearly there are countless more successful drugs and trials than there are questionable one. Lets not cost yet more lives by dramatic sensationalism and spooking the herd into a stampede of not taking drugs that do actual prolong and vastly improve people's lives.
Reply | Report Abuse | Link to thisI think a good start-up read to this book, one that I've already read and as a student of the pharmaceutical sciences, thoroughly enjoyed, was "Body Hunters" by Sonia Shah.
Reply | Report Abuse | Link to thisIt would seem to me that the high rate of positive results from pharma drug trials is simply because they are only willing to invest in a trial when they are already confident of a positive result.
Reply | Report Abuse | Link to this