## How to be uncertain about a therapy’s effects without being inconsistent

A clinician contemplating the use of a therapy faces one of the most difficult mental tasks: the weighting of external evidence (in the form of published scientific literature, product labels and testimonies from colleagues) against one’s personal opinion (whether this is grounded in previous experience, specific scientific understanding or other poorly differentiated factors).

The external evidence can take many forms, but in the case of therapies backed by hard outcomes data it will usually be summarized in the form of frequency (percentage of successes, $f$) over a large number of patients ($N$) who participated in a number of systematic evaluations, e.g. Randomized Controlled Trials, registries or industry-sponsored post-authorization studies).

On the other hand the personal opinion is more likely to be un-differentiated, with the possible exception of previous experience in which an astute clinician may draw from a personal series of successes and failures. For a new therapy, such experience may not even be available leaving the clinician entirely uncertain about the success rate of what’s about to be prescribed.

So how can one combine these diverse sources into an internally consistent evaluation that is practical and quantitative so that the expectations of success can be made explicit to patients?

As I have said in a previous post adoption of a Bayesian perspective is all that’s required to simultaneously achieve:

integration of diverse sources of evidence with prior opinion
avoidance of inconsistencies
quantitative practicality

To illustrate Bayesian reasoning we will assume the simplest case of a clinician who has just read this great paper mentioned at the beginning of the post but without any previous experience in which to ground his prior opinion.
Striving to maintain maximum impartiality, favoring neither optimism nor pessimism about the therapy what is the prior probability that the clinician should use? An assignment that is often use is for the clinician to augment the actual number of successfully treated patients in the trial, $S$, and the number of failures, $F$, with fictional pseudo-observations that express such maximum impartiality (or ignorance).

The usual conjugate assignment is the use of the Beta distribution with $s$ and $f$ number of pseudo-successes and failures. The three uninformative priors that can be used to express maximum impartiality under different states of ignorance are:

• The Haldane prior with $f=s=0$, when one is not even certain whether both success or failure are possible (eg the therapy can save or kill everyone)
• The Jeffrey’s prior with $f=s=1/2$ which expresses ignorance in both the odds scale
• The rule of succession prior $f=s=1$ advocated by Laplace

Irrespective of which prior is used, after obtaining the trial data one updates his belief about the probability of success to:
$\frac{S+s}{S+s+F+f}$

This formula shows the overwhelming impact the trial data can have over the prior; for a trial with 40 successes and 60 failures, the three priors yield the following expectations for the success rate: 40%, 40.09% and 40.19% respectively.

With such conjugate prior assignments, one can take into account one’s personal experience by simply adding one’s successes, $o_s$ and failures $o_f$ to yield the following expression for the expectation of the success rate:
$\frac{S+s+o_s}{S+s+o_s+F+f+o_f}$

The aforementioned formula immediately suggests one reason for our trust for the result of a large trial over our experience and why we would not be swayed much by a small to moderate trial in an area in which we have a large personal experience. In the former case the fraction is determined by thew number of successes and failures in the trial, whereas under the second scenario our own experience dominates the data from the study.