How to be uncertain about a therapy’s effects without being inconsistent

A clinician contemplating the use of a therapy faces one of the most difficult mental tasks: the weighting of external evidence (in the form of published scientific literature, product labels and testimonies from colleagues) against one’s personal opinion (whether this is grounded in previous experience, specific scientific understanding or other poorly differentiated factors).

The external evidence can take many forms, but in the case of therapies backed by hard outcomes data it will usually be summarized in the form of frequency (percentage of successes, f) over a large number of patients (N) who participated in a number of systematic evaluations, e.g. Randomized Controlled Trials, registries or industry-sponsored post-authorization studies).

On the other hand the personal opinion is more likely to be un-differentiated, with the possible exception of previous experience in which an astute clinician may draw from a personal series of successes and failures. For a new therapy, such experience may not even be available leaving the clinician entirely uncertain about the success rate of what’s about to be prescribed.

So how can one combine these diverse sources into an internally consistent evaluation that is practical and quantitative so that the expectations of success can be made explicit to patients?

As I have said in a previous post adoption of a Bayesian perspective is all that’s required to simultaneously achieve:

      integration of diverse sources of evidence with prior opinion
      avoidance of inconsistencies
    quantitative practicality

To illustrate Bayesian reasoning we will assume the simplest case of a clinician who has just read this great paper mentioned at the beginning of the post but without any previous experience in which to ground his prior opinion.
Striving to maintain maximum impartiality, favoring neither optimism nor pessimism about the therapy what is the prior probability that the clinician should use? An assignment that is often use is for the clinician to augment the actual number of successfully treated patients in the trial, S, and the number of failures, F, with fictional pseudo-observations that express such maximum impartiality (or ignorance).

The usual conjugate assignment is the use of the Beta distribution with s and f number of pseudo-successes and failures. The three uninformative priors that can be used to express maximum impartiality under different states of ignorance are:

  • The Haldane prior with f=s=0, when one is not even certain whether both success or failure are possible (eg the therapy can save or kill everyone)
  • The Jeffrey’s prior with f=s=1/2 which expresses ignorance in both the odds scale
  • The rule of succession prior f=s=1 advocated by Laplace

Irrespective of which prior is used, after obtaining the trial data one updates his belief about the probability of success to:
\frac{S+s}{S+s+F+f}

This formula shows the overwhelming impact the trial data can have over the prior; for a trial with 40 successes and 60 failures, the three priors yield the following expectations for the success rate: 40%, 40.09% and 40.19% respectively.

With such conjugate prior assignments, one can take into account one’s personal experience by simply adding one’s successes, o_s and failures o_f to yield the following expression for the expectation of the success rate:
\frac{S+s+o_s}{S+s+o_s+F+f+o_f}

The aforementioned formula immediately suggests one reason for our trust for the result of a large trial over our experience and why we would not be swayed much by a small to moderate trial in an area in which we have a large personal experience. In the former case the fraction is determined by thew number of successes and failures in the trial, whereas under the second scenario our own experience dominates the data from the study.

Advertisements

Tags: ,

One Response to “How to be uncertain about a therapy’s effects without being inconsistent”

  1. Four basic attitudes towards efficacy and its relation to effectiveness | Statistical Reflections of a Medical Doctor Says:

    […] is about probability and stats as applied to solve patient problems and other stuff! « How to be uncertain about a therapy’s effects without being inconsistent Illussion of Effectiveness in the ‘definitive’ clinical trial […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: