On the futility of discourse between two people of different degrees of conviction : Reflections on the Lindley Paradox

“In an exchange between two people, it is the one with the stronger conviction,not the better grasp of reality, who will win.“

The phrase above snapped into my head when I watched an honest and very extensive dialog between two business partners who were examining an issue. Partner A had come into the table with a very strong conviction about the issue and what needed to be done, while partner B was not so sure and was open to entertain a number of possible courses of action. Over a course of 45 min, they examined the data available to them and decided to proceed with A’s opinion. What is surprising is not the decision per se , but the fact that before the meeting A had confided to me that the available information contradicted his belief; B had also said that A’s position was rather implausible given the same data. Yet, when they sat down and considered what was in front of them they reached a rather different conclusion!

This situation got me thinking about similar encounters I had witnessed over the years: case discussions in the clinic, interpretation of experimental data during lab meetings or even something closer to home ie conversations I’d had with my wife. Recounting such cases led me to conclude that in these contexts, which compare and contrast different narratives, the winner was almost always the one walking in the debate with the stronger conviction. Discounting alternative explanations of intimidation, deference or unwillingness to challenge a figure of authority, uneven intellectual capacity or even greater determination (which were minimal in the cases I considered) this is a surprising observation. It becomes even more interesting when I reflected back on the cases for which I had additional insights about the participants’s individual appraisal of the situation. In these, the decision that was collectively reached was the one of the party with the stronger conviction, which however had been individually deemed unsuitable when each of the parties had examined it on their own.

This pattern rang a bell in that it recapitulates a well known statistical paradox by Lindley : one obtains a piece of datum that would ordinarily lead to the rejection of a null hypothesis assumed to be true (the equivalent of a the person with the strong conviction considering the situation on their own). Yet, when this strong conviction is considered relative to other possible evaluations it comes to emerge as a winner, if the alternative hypotheses considered are too many. The paradox in this case emerges when one considers the alternative hypotheses on their own (Wikipedia has a nice numerical example about the number of births of boys vs girls); this evaluation would lead one to conclude that the data under the null hypothesis are highly unlikely, yet when a direct comparison of the null and the alternatives is carried out, similar to the one considered here, the null emerges as the indisputable winner. As the article in Wikipedia says, there is no actual paradox here, only the failure to realize that rejection testing of hypotheses is a different problem from their comparative evaluation. In the former, we seek to examine whether the hypothesis provides a poor explanation for the situation at hand/data, whereas in the former we find that this “strong” hypothesis provides a much better explanation than the alternatives. In the literature there have been other attempts to explain the paradox (Robert’s article gives a short historical overview, while a powerpoint version for problems in High Energy Physics may be found here). Some Bayesians even dismiss Bayesian hypothesis testing (Gelman has been arguing this point in his books and blog) in favour of evaluation of hypotheses similar to the path the wikipedia article takes to explain the paradox, while others demolish the idea of hypothesis testing from either perspective.

When one delves into the mathematics, one will discover that the crucial factor is the large asymmetry between the convictions of the two hypotheses (or the parties involved in my encounters): one walks into the table with a strong predetermination and the other one sponsors a very weak conviction about the state of affairs. This asymmetry leads to the paradoxical situation of having to accept the stronger conviction, even though the empiric data are highly improbable assuming that this narrative of the world is true. Or as Robert puts it: ” being completely indecisive about the alternative … means we simply should not chose it.”

What are the implications of this paradox for the non-statistician? Part II has the answers!

Tags: Bayesian, hypothesis, Lindley paradox, statistics