Four basic attitudes towards efficacy and its relation to effectiveness

I will continue this series of posts regarding the appraisal of efficacy (“how well a therapeutic intervention worked in a randomized experiment”) and its translation to statements about effectiveness (“how well the intervention worked in the real world”), by considering the attitudes that one adopt towards these issues. The aim is to develop a sophisticated approach, or rather a vantage point that one would almost always want to adopt when considering the implications of having data about the efficacy (results of a trial) and effectiveness (success rate in real world practice). However the vantage point will only become evident by considering a basic set of attitudes, which are described here:

  • The believer is the individual to whom the evidence from the trial reins supreme, and that trial results translate perfectly in the real world. In other words, this individual takes the position that the aforementioned therapy is as effective as efficacious. The believer thus assumes that real world patients and study participants are exchangeable so that he or she will apply the Bayes theorem as shown here, adding together the successes (and failures) from the real world and the trial to gain a precision premium
  • The sceptic, who takes the position that effectiveness is less than efficacy, but cannot (or will not) say by how much. This is similar to the position presented in previous posts in this blog (here, here while the mathematics can be found here)
  • The enthusiast who views the trial results only as the proof of concept that THIS therapy can change the world. To this individual widespread adoption of the intervention under question can change the world, as effectiveness is surely likely to be much higher in everyday practice. Thus the attitude of the enthusiast is the opposite of the skeptic: given the chance, the enthusiast would put this therapy into the water to cure as many people as possible
  • The agnostic is the person who views the trial results as providing evidence that the intervention is neither rat-poisson nor holy water, that kills or cures respectively, everyone who takes it. To this person, the take home point from the trial is that the therapy works in some, and fails in others but the percentage of responders defining effectiveness doesn’t quantitatively translate in everyday practice. Hence the agnostic, has no use of efficacy trials (except possibly as providing evidence for relative efficacy) and would base his assessment of effectiveness only on real world results. Quantitatively this person attitudes is summarized in Laplace‘s rule (or even the Jeffrey‘s prior if the agnostic prefers odds over probabilities)

All four individuals are assumed to have internally consistent reasoning powers (and thus are Bayesians in both spirit and practice), so if they found to differ in their appraisal, this difference can solely be attributed to the difference in their attitude. To contrast this four attitudes we will run a simple thought experiment in which these four people are given the results of a randomized trial, and are asked to quantify their assessment of effectiveness. For illustration purposes, we will assume that success was observed in 50% of the 1000 trial participants exposed to it (a moderately sized trial) and ask for the best guess of effectiveness under three different scenarios: 20%, 50%, 60%. The real world data will come in very different sizes, consisting of the success rate in :

  • 20 patients (typical of a single clinician’s experience in the first few months of the adoption of the intervention)
  • 100 patients (typical of the experience of a single practice with many partners)
  • 300-800 patients (typical of the sample size employed in industrial post-authorization observational studies)
  • 1000-5000 patients (typical of regional or small country registries)
  • 10000-20000 patients (typical of larger, possibly multicountry registries)

Before considering the specifics, a personal disclaimer is due: over the years my attitude has substantially changed, so what the enthusiast-me (late years in medical school), was transformed into a believer-me by indoctrination into Evidence Based Medicine (residency/fellowship). In the last few years I have practised outside an academic medical center I have become somewhat more of a skeptic when it comes to using a therapy, as opposed to reading about it, while my agnostic side dominates when doing research or measuring quality. Hence, there is a practical importance in evaluating alternative attitudes as they relate to the different roles of a physician over his or her formative years. Having laid out the conditions of the experiment, we will turn to the evaluation of these four attitudes over the next few posts


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: