By Heidi Ledford of Nature magazine
An analysis of nearly three-dozen highly cited papers has found that researchers often overstate the link between biomarkers and disease by citing papers that report the strongest association, even when subsequent analyses downplay the connection.
Biomarkers are biological characteristics, such as the activity of a gene or protein, which can be used to monitor a person's health. They are key to the success of personalized medicine: biomarkers may predict whether a person is likely to develop a disease and how they will respond to a given treatment.
But researchers have struggled to develop reliable biomarkers, and the field is riddled with biological tags that initially showed promise, only to crumble under further scrutiny.
"There is a huge literature, with thousands of studies being published every year and with lots of highly promising claims being made in prestigious journals," says John Ioannidis, an epidemiologist at Stanford University in California. "Yet very few make it to the clinic."
Now, in a study published today by the Journal of the American Medical Association, Ioannidis and his colleague Orestis Panagiotou of the University of Ioannina in Greece, show that researchers often perpetuate the hype surrounding a biomarker even after larger studies, or meta-analyses of multiple studies, have undercut its significance.
"This is not just anecdotal," agrees Patrick Bossuyt, an epidemiologist at the University of Amsterdam in the Netherlands who was not involved in the study. "It occurs very frequently."
Ioannidis has himself published papers linking promising biomarkers to disease, only to learn later that the association failed to hold up. The experience, he says, sensitized him to the need for rigorous testing.
In the current analysis, Ioannidis and Panagiotou focused on a set of two-dozen frequently cited biomedical journals. They selected biomarker papers that had been cited more than 400 times, and then narrowed the group down to those that had been included in a subsequent meta-analysis incorporating multiple studies of that same biomarker. These meta-analyses provided a way to compare the highly cited paper with other studies.
Of the 35 studies that met their selection criteria, 29 demonstrated a stronger link between biomarker and disease than the subsequent meta-analysis. And 30 reported a stronger association than observed in the largest single study of the same biomarker.
For example, a 1991 study that was cited 1,436 times found that patients with a high level of a compound called homocysteine in their blood had a 27.7-fold elevated risk for vascular disease. But a meta-analysis reported only a 1.58-fold increased risk.
"Many investigators are citing the studies that have the most optimistic results," says Ioannidis. "There's clearly a very strong citation bias."
That is probably so, agrees Bossuyt. But, he adds, the study may have inflated the frequency with which this occurs.
Bossuyt notes that researchers often conduct meta-analyses when studies yield conflicting results. By selecting only those studies that received this treatment, Ioannidis and Panagiotou may have biased their sample towards biomarkers that did not hold up to subsequent testing.
Furthermore, Bossuyt notes that in some cases the highly cited study may have used a different patient population than other studies of the same marker. For example, an experiment comparing a biomarker in healthy controls versus patients with advanced Alzheimer's disease may find a stronger link than one involving patients with only mild cognitive impairment.
Nevertheless, Bossuyt does not doubt that researchers are biased both in what they publish and what they cite: "In all of science, there is an emphasis on the positive and the surprise.
This article is reproduced with permission from the magazine Nature. The article was first published on May 31, 2011.