As we shall shortly see, this problem of unpublished data is widespread throughout medicine, and indeed the whole of academia, even though the scale of the problem, and the harm it causes, have been documented beyond any doubt. We will see stories on basic cancer research, Tamiflu, cholesterol blockbusters, obesity drugs, antidepressants and more, with evidence that goes from the dawn of medicine to the present day, and data that is still being withheld, right now, as I write, on widely used drugs which many of you reading this book will have taken this morning. We will also see how regulators and academic bodies have repeatedly failed to address the problem.
Because researchers are free to bury any result they please, patients are exposed to harm on a staggering scale throughout the whole of medicine, from research to practice. Doctors can have no idea about the true effects of the treatments they give. Does this drug really work best, or have I simply been deprived of half the data? Nobody can tell. Is this expensive drug worth the money, or have the data simply been massaged? No one can tell. Will this drug kill patients? Is there any evidence that it’s dangerous? No one can tell.
This is a bizarre situation to arise in medicine, a discipline where everything is supposed to be based on evidence, and where everyday practice is bound up in medico-legal anxiety. In one of the most regulated corners of human conduct we’ve taken our eyes off the ball, and allowed the evidence driving practice to be polluted and distorted. It seems unimaginable. We will now see how deep this problem goes.
Why we summarize data
Missing data has been studied extensively in medicine. But before I lay out that evidence, we need to understand exactly why it matters, from a scientific perspective. And for that we need to understand systematic reviews and “meta-analysis.” Between them, these are two of the most powerful ideas in modern medicine. They are incredibly simple, but they were invented shockingly late.
When we want to find out if something works or not, we do a trial. This is a very simple process, and the first recorded attempt at some kind of trial was in the Bible (Daniel 1:12, if you’re interested). First, you need an unanswered question: for example, ‘Does giving steroids to a woman delivering a premature baby increase the chances of that baby surviving?’ Then you find some relevant participants, in this case, mothers about to deliver a premature baby. You’ll need a reasonable number of them, let’s say two hundred for this trial. Then you divide them into two groups at random, give the mothers in one group the current best treatment (whatever that is in your town), while the mothers in the other group get current best treatment plus some steroids. Finally, when all two hundred women have gone through your trial, you count up how many babies survived in each group.
This is a real-world question, and lots of trials were done on this topic, starting from 1972 onwards: two trials showed that steroids saved lives, but five showed no significant benefit. Now, you will often hear that doctors disagree when the evidence is mixed, and this is exactly that kind of situation. A doctor with a strong pre-existing belief that steroids work—perhaps preoccupied with some theoretical molecular mechanism, by which the drug might do something useful in the body—could come along and say: “Look at these two positive trials! Of course we must give steroids!” A doctor with a strong prior intuition that steroids were rubbish might point at the five negative trials and say: “Overall the evidence shows no benefit. Why take a risk?”
Up until very recently, this was basically how medicine progressed. People would write long, languorous review articles—essays surveying the literature—in which they would cite the trial data they’d come across in a completely unsystematic fashion, often reflecting their own prejudices and values. Then, in the 1980s, people began to do something called a “systematic review”. This is a clear, systematic survey of the literature, with the intention of getting all the trial data you can possibly find on one topic, without being biased towards any particular set of findings. In a systematic review, you describe exactly how you looked for data: which databases you searched, which search engines and indexes you used, even what words you searched for. You pre-specify the kinds of studies that can be included in your review, and then you present everything you’ve found, including the papers you rejected, with an explanation of why. By doing this, you ensure that your methods are fully transparent, replicable and open to criticism, providing the reader with a clear and complete picture of the evidence. It may sound like a simple idea, but systematic reviews are extremely rare outside clinical medicine, and are quietly one of the most important and transgressive ideas of the past forty years.
See what we're tweeting about
18 Comments
Add CommentThis is an interesting commentary and it certainly has some merit in pointing out deficiencies in our current system of clinical trials. However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".
Reply | Report Abuse | Link to thisIt is likely that the trials sponsored by pharmaceutical companies are better funded, larger, and more carefully controlled. These trials are often admittedly designed to show efficacy - sometimes in a narrow population - as opposed to effectiveness. You simply have to do this to get a drug approved. There is nothing evil or inappropriate in this. If it doesn't work in a focused population, you might as well stop there. So, success and failure, independent versus pharma, need to be studied in similar trials otherwise you are comparing apples to oranges and committing the same sins of bias alluded to here. This is very difficult to do as pharma trials tend to be very different than independent trials in design, scope, and patient population. As to meta-analysis, namely, the analysis of a compilation of studies, this is extremely error prone and subject to bias and selection error as the author well knows.
Independently funded research is often conducted with an agenda no less biased than that of pharmaceutical companies. There are many people who set out with a goal, conscious or unconscious, of proving that a drug does not work and design their trials accordingly. For some, it seems good sport to be a thorn in the side of industry.
The truth is that nothing is perfect and perfect is the enemy of the good. There have been staggering R&D cuts in the pharmaceutical industry throughout the world due in large part from regulatory hurdles that demand clinical trials of unsupportable size, cost, and duration. This can chew up so much patent life and capital that companies simply give up.
Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research.
@deneb
Reply | Report Abuse | Link to this"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem""
It's clear he has the right kind of evidence to support his conclusions so I don't worry about tone and style.
Hi there,
Reply | Report Abuse | Link to thisI wrote the book that's extracted above.
"I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem"."
There is a discussion at the end of each section, and the end of each chapter, and then at the end of the book, about what can be done to remedy these problems.
There is also a campaign at www.alltrials.net for the results of all trials on all currently used treatments to be reported in full, that has been signed by over 30,000 people, 80 patient groups, research funders, medical bodies, and the drug company GSK, who have signed up to our commitment:
http://www.badscience.net/2013/02/this-is-excellent-and-amazing-gsk-have-just-signed-up-to-alltrials-net/
Do please sign yourself:
www.alltrials.net
"Considering the poor trajectory of the pharmaceutical industry today, the pundits may get what they want; few industry funded trials because there will be little financial incentive and even less research."
I wouldn't want that, nor do I ask for it. I simply ask that all results for all trials on all treatments used by patients are made available. It is impossible for doctors and patients to make informed decisions when results are withheld. Industry should be able to make money from good treatments (as they often do) without hiding information on them. I don't see that this is unreasonable, and you are also incorrect to assume that these issues only affect industry: academic studies are also frequently withheld, as I have discussed at length here and elsewhere.
Like many of the issues in the book, these issues are structural, and they are long overdue fixing.
at bengoldacre
Reply | Report Abuse | Link to thisThank you.
Most of normal us have this altruistic view, big pharma et al, like to hide everything so people don't get to see the real picture.
I signed your petition. :)
Shame we cant edit our posts to remove grammar errors though, as we race to type things.
Reply | Report Abuse | Link to this@Ben
Reply | Report Abuse | Link to thisThough I appreciate your campaign for openness, how are you planning to address the significant intellectual property rights aspect of funded research? It is a sticky issue, but don't you think that investors have the right to discover valuable information without giving it away to everyone, including their competitors?
Though patients and doctors need to make informed decisions, they are buying products from companies that have invested a great deal of money and skill into developing treatments that are often very easy and materially inexpensive to copy. Patents provide a limited amount of protection against the unfair competition that results when copycats have equal access to information.
I admit that I am troubled by the current situation and wish that information was more freely available. As taxpayers, we have the ability to ask our government to spend more of our money on publicly funded research, but our constitution is based on the principle that we do not have the right to take other people's property for public good without fair compensation.
Rod Adams
Publisher, Atomic Insights
Drug companies also undermine their own research by not disclosing what's in their placebos (and active placebos). They're allowed to manufacture their own placebos, some of which produce dry mouth and other symptoms (by design) and not report the ingredients. I have written about this recently in my blog at http://asserttrue.blogspot.com/2013/01/are-placebos-really-sugar-pills.html.
Reply | Report Abuse | Link to thisThank you for a great article.
Possible corrections to my comment:
Reply | Report Abuse | Link to this>However, I think the author shows a clear bias against pharmaceutical research in his tone and style and would be surprised to see a conclusion other than this is a "problem".<
1) It's clear he has the right kind of evidence to support his conclusions so in this case I don't worry about tone or style.
2) I agree tone and style are important, but in this case they are so tame and the evidence is so strong there really is nothing to complain about.
3) I think the author does not show a 'clear bias against pharmaceutical research' but rather he shows 'clear and solid evidence against certain research practices in the pharmaceutical industry'.
Are you insane? Do you read your comments before posting them? When was the last time you checked your ethimometer?
Reply | Report Abuse | Link to thisI agree that all drug or therapy research has an agenda regardless of its funding source. However the gravy train of industry funded research is greased by mining for positive results regardless of how inane. While it is not fraud, it does create a lean to research literature that researchers, clinicians and others trained in the business recognize and defend against. This is one of the reasons pharmaceutical companies moved to direct consumer advertising. It is easier to create demand with images of happy families running through flower draped fields than by producing research papers that are going to be reviewed with a jaundiced eye by the prescriber.
Reply | Report Abuse | Link to thisAh, so! Those industry trials and surveys, are just as biased, as the ones, that I took, and often did not complete, because despite my answers, I was being led towards a conclusion, I could not support.
Reply | Report Abuse | Link to thisDitto political ones that tend to lead to a conclusion, the paid surveyor, promised his client.
No surprise here. The pharmaceutical industry (and chemical industry as a whole) have been cherry picking results of studies for decades. As far back as the early 1980s I wrote a series of stories about "cherry picking" that was so blatant that it was considered fraud in the testing of hundreds of food additives, drugs, pesticides, and industrial chemicals.
Reply | Report Abuse | Link to thisIndustrial Bio-Test Laboratories (IBT) was a particularly prolific source of toxicological testing. It also was involved in massive misconduct that resulted in the indictment and conviction of its president and a number of top executives in 1983. An audit by the FDA found that 71% of 867 studies were invalid because of "numerous discrepancies between the study conduct and data".
However, IBT was not an isolated case as the pharmaceutical and chemical industries claimed. Information I gathered from the FDA in 1980-1982 revealed that more than two dozen other labs had also engaged in significant levels of misconduct.
Three decades have passed and not much has changed when it comes to ensuring the integrity of safety studies on a massive number of products that we are exposed to.
As we move into an era in which biotechnology and nanotechnology will have an increased impact on our lives, the integrity of safety studies needs to be beyond question. Unfortunately, the lack of transparency makes it all to easy to continue to high unfavorable results.
AllTrials is a good start, but we need to ensure that not only results, but all of the data and study documentation are made available online so that many eyes can verify independently that "cherry picking", selective reporting, or outright fraud and fakery are not happening.
I'd say this has bias, I've seen an antidepressant drug research discontinuation seriously considered, don't konw what happened in the end, because more suicide attempts, although not reaching statistical signification, were in the experimental than in the comparator group. For pre-market trials, aimed to obtain a drug registration, this simply can't happen, Health Regulatory Authorities watch over all the process, and no fake data can be presented. Post-marketing studies are a different thing, as long as an study has not a Registered Trial Number, you cannot trust it, but Drug Surveillance data about unexpected side effects, not known in the early phases, when not too many patients had received the drug are even harder to hidden, as the economical consequences of hidding data are so enormous, that no company can seriously thinking it can be afforded, and a double registry, of drug producer and HRAs exists. There was another book on this more tan 20 years ago: "Dangerous medicine", I received it as a tip gift from a very nice person, but haven't found yet the time to read it, reality is often much more interesting than fiction.
Reply | Report Abuse | Link to thisA good read and well presented. Sadly, not very surprising findings given what I know about business and humans. You really can't help wanting to find positive results for the company you work for, where as an independent comes in with a skeptical eye.
Reply | Report Abuse | Link to thisThe details about the consequences of unpublished research are clear and important. It's unfortunate that the author starts with the statistics about how many studies support the drug being tested, because this is exactly the result you would expect if drug companies were doing what we want them to do in a highly ethical way.
Reply | Report Abuse | Link to thisLook, studies are expensive, so your "ideal" drug company is going to to do large, repeated studies on drugs they believe actually work. That is, they will re-test, in larger populations, the drugs that showed promising reults in small tirals. Unless drug companies are quite incompetent, the later trials will tend to confirm the ealier trials, producing exactly the result that the author presents as evidence of misconduct: most trials deonstrate the effectiveness of the drug being tested.
This lapse does not impugn the importance of the story he tells later about missing information in unpublished studies.
Clearly the past 'industry study' negative bias towards pharma companies has stopped with this article; Not. I agree that a case appears to have been made for greater transparency, but how would we know for sure with such blatant negative bias. Clearly there are countless more successful drugs and trials than there are questionable one. Lets not cost yet more lives by dramatic sensationalism and spooking the herd into a stampede of not taking drugs that do actual prolong and vastly improve people's lives.
Reply | Report Abuse | Link to thisI think a good start-up read to this book, one that I've already read and as a student of the pharmaceutical sciences, thoroughly enjoyed, was "Body Hunters" by Sonia Shah.
Reply | Report Abuse | Link to thisIt would seem to me that the high rate of positive results from pharma drug trials is simply because they are only willing to invest in a trial when they are already confident of a positive result.
Reply | Report Abuse | Link to this