The study and application of health outcomes research to the management of pharmaceuticals is a messy business. Research tools range from the large randomized controlled trial to the small, self-reported health outcomes study. Considerable uncertainty still exists about the best methodology for many areas of inquiry, and commercial interests and publication bias runs rampant. While pharmaceutical manufacturers are the most studied offenders, health care vendors are all potential violators; and of course, bias is not limited to those with commercial interests.
A study published earlier this year in JAMA once again brought awareness to the extent of the bias problem, with headlines reporting “Science for Sale.” Considering top medical journals, scientists reviewed over 600 studies from 2006 that had reported statistically non-significant primary outcomes. They subsequently conducted a detailed analysis of 72 of those they considered to be of the highest quality—all randomized controlled trials. Upon detailed review, they found that 50 percent of the articles misrepresented the data in the study conclusions, leaving the impression that the treatments were effective even though the primary results indicated otherwise. This “spin” —which the study authors define as specific reporting strategies, whatever their motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results—also appeared in nearly 60% of the conclusions found in the study abstracts.
If the conclusions in 50 percent of the studies published in top medical journals are being spun toward independent interests, what is the magnitude of distortion in health outcomes research related to pharmaceuticals, where the methodological choices are greater, the standards less well-defined, the chance of study registration prior to initiation far less likely, and the quality of peer-review often suspect? The issue of bias only serves to compound decision-makers’ challenge in reviewing and applying the rapidly growing body of health outcomes research to make a well-informed decision about their pharmacy benefits. Recognizing this challenge, in the months ahead, our plan is to laud those who use the right evaluation methods for health outcomes assessment and call out those who do not and to provide simple tools for decision-makers to increase their knowledge and ability to see through the rhetoric. Finally, by adding another voice on the side of rigorous analysis, the truth about what works and what doesn’t will continue to crowd out studies that are merely repackaged marketing.