Suppose one is approached by one’s partner with the results of a new intervention that helped X% of N carefully chosen participants in a Randomized Controlled Trial (RCT), with only minimal adverse events (seen in Y% of the N patients). The colleague, a TRUE BELIEVER – champion of innovation and defender of progress against the medical luddites of this world, wants to convince you to implement this new therapy as part of the standard protocol in your common practice. He is thinking that it should be offered to all newcomers, including patients who would not have been eligible to participate in the aforementioned trial. What would a healthy sceptic do? Champion innovation and adopt the new therapy on the spot, or defend tradition and wait? Is it possible to ground the answer in the cold, objective language of math and warm up to/cool down your partner accordingly?
To approach this problem, we have to translate the sceptic’s attitude towards new therapies into math: the new intervention will work less effectively and will associated with more complications than the trial experience suggests. In other words, the sceptic thinks that the real world effectiveness will be smaller than the trial efficacy, while things will not go as smoothly as “advertised” in the peer review article. This sceptic position is not at all unjustified: things will always work better in the controlled settings of a trial (or any experiment for that matter). Trials, being tuned signal detection devices by design, will almost invariably have excluded the patients least likely to benefit or the ones more likely to suffer harm (for example most trials exclude patients with kidney or liver impairment for safety reasons!). On the other hand, certain real world constraints may not be operable in a trial setting, a situation that is reminiscent of the distinction between science and engineering and the effort required to turn scientific discoveries into technological advances. In both situations (science/RCT vs. engineering/medical practice) what will work in the controlled setting may never work as well (or without extra effort) in the real world, calling for at least some scepticism when translating research findings into practical innovations.
The other element that need to be translated into mathematical (or rather probabilistic) statements are the success/failure and complication/sail-through rates observed in the trial, as these are the only objective data available to believers and sceptics alike. In the absence of additional information, the standard approach is to use a beast known as the Beta distribution to encode one’s belief about a given percentage or proportion having observed a number of successes and failures, providing thus the first step towards a formal quantitative treatment. Cranking out the math that models an agnostic sceptic’s belief about his scepticism leads to the following statement:
- in the absence of real world data, the skeptic’s expectation about the effectiveness of an intervention in the real world is half the efficacy observed in the RCT.
So if the intervention helped 60% of the participants in the trial (efficacy = 60%) , the expected real world effectiveness is only 30% (things are even worse for safety anticipations and this is left as an exercise for the reader). This situation represents an interesting conundrum for both the sceptic and the believer, with the former expecting only 50% of the benefit the latter sees, even though both have been presented the same randomized trial data. There are also implications for the other stakeholders in the health care system:
- patients who will have to consider the new therapy for their care, weighing risk, benefit and out-of-pocket expenses
- third party payors , who will ultimately decide, or not to cover the therapy) and make appropriate cost sharing/reimbursement decisions
- regulatory authorities, who will have to consider approval of the therapy for use in the real world based on RCT data)
- healthcare companies who have spent an enormous amout of resources to develop the new therapy and will market it post regulatory approval to physicians (and depending on jurisdiction even patients)
Each of this parties may adopt an enthusiastic/true-believer, an agnostic sceptic perspective or even an intermediate position leading to diversity of opinion and action even though everyone has evaluated the same data. So how can consensus ever be reached? Do comparative evaluations matter in this setting and how (much)?
The case is open for discussion ….