In a thoughtful commentary published in the British Medical Journal, clinical researchers from Europe question the claims of cost-effectiveness made for many commonly used pharmacological treatments. The authors argue that “…although there are claims that important preventive drugs such as statins, antihypertensives, and bisphosphonates are cost effective,6 7 8 9 there are no valid data on the effectiveness, and particularly the cost effectiveness, in usual clinical care. Despite this dearth of data, the majority of clinical guidelines and recommendations for preventive drugs rest on these claims.”
The authors cite a 2009 study, which examined a cost-effectiveness model of selective cyclo-oxygenase-2 (COX 2) inhibitors, as evidence of the weak external validity of claims of cost-effectiveness. The COX-2 evaluation, which was based on a clinical trial, found that the cost of avoiding one adverse gastrointestinal event by switching patients from conventional non-steroidal anti-inflammatory drugs to COX-2 inhibitors was approximately $20,000. In contrast, when the same analysis was conducted using the UK’s General Practice Research Database, which includes patients’ medical records in routine care, the cost of preventing one bleed was over $100,000.2
These findings are similar to work myself and others conducted several years ago on the COX-2s. While the original cost-effectiveness model for the U.S. reported a cost per year of life saved (YLS) of about $19,000 for COX-2s when compared to non-selective NSAIDs, our revised model, which was based on actual practice, found a cost per YLS of $107,000.
In a different study, we examined the external validity of a cost-effectiveness model of treatment options for eradication of h. pylori. The original decision-analytic model found that the lowest cost per effectively treated patient was for the dual combination of PPI and clarithromycin ($980), whereas we found that the lowest cost per treatment was for the triple combination of bismuth, metronidazole, and tetracycline at a cost of $852. Why the disconnect? In the original h. pylori model, the authors had made assumptions about medication compliance and the cost of recurrence that simply did not hold up in the real-world.
In the case of the COX-2s, the recent commentary concluded that the published cost-effectiveness analyses of COX 2 inhibitors neither had external validity nor represented the patients treated in clinical practice. They emphasized that external validity should be an explicit requirement for cost effectiveness analyses that are used to guide treatment policies and practices. At least one academic modeler vehemently disagrees with the requirement of external validity, arguing that “it is wrong to insist that models be ‘validated’ by events that have not yet occurred; after all, the modeler cannot anticipate advances in technology, or changes in human behavior or biology. All that can be expected is that the model reflects the current state of knowledge in a reasonable way, and that it is free of logical errors.”
It is true that right when a drug comes to market, the only available data will likely be from the original clinical trials used to seek FDA approval, and the modeler will be forced to make numerous assumptions about compliance, costs, concomitant medication use, etc. The problem is that the extent to which these assumptions are made without bias is unclear. Research has shown that sponsorship by the pharmaceutical industry affects the results of economic models. In a review published in 2010, researchers found that 95 percent of studies sponsored by pharmaceutical manufacturers reported favorable conclusions compared to only 50 percent of nonsponsored studies. While it could be argued that this reflects a publication bias, the validation studies that I have described above suggest otherwise. In each of these cases, there were key assumptions that drove model outcomes which, from a plan sponsor perspective, we found highly questionable at the time the model was first published.
Surprisingly, the issue of model validity receives relatively little attention given the central role that these models play in the field of pharmacoeconomics, as for example, in the AMCP dossier process. The commentary authors argue that real-world comparative studies are the key to producing cost-effectiveness models that possess external validity. This certainly will help with the quality of models post-FDA approval. However, for models used at the time a drug is launched, ultimately, I expect that plan sponsors will have to develop their own models to ensure systematic bias is removed.