Likelihood Principle

What constitutes evidence in science? When we collect data, does its meaning depend solely on the numbers themselves, or also on our intentions for collecting them? This question lies at the heart of a profound and controversial idea in statistics and the philosophy of science: the Likelihood Principle. This principle offers a clear-cut answer, suggesting that the intentions behind an experiment, such as the decision of when to stop collecting data, are irrelevant to the evidence once the data is in hand. However, this seemingly simple idea creates a fundamental schism in statistical practice, challenging the foundations of many widely used methods.

This article delves into this pivotal concept. In the first section, ​​Principles and Mechanisms​​, we will unpack the formal definition of the Likelihood Principle through its core component, the likelihood function, and illustrate the central conflict it creates between frequentist and Bayesian approaches. Subsequently, in ​​Applications and Interdisciplinary Connections​​, we will explore the real-world battlegrounds where this principle matters most, from clinical trial design to the validation of artificial intelligence, revealing its deep implications for how we learn from data.

Imagine you are a detective at the scene of a crime. On a table, you find a single, crucial clue: a handwritten note. Your task is to infer the identity of the person who wrote it. Two scenarios could have led to this discovery. In the first, you were told to search the house for exactly one hour, and in that time, you found only this one note. In the second, you were told to search until you found the first piece of evidence, and this note was it.

Now, a question that gets to the very heart of scientific reasoning: Does how you were instructed to search change the evidential meaning of the note itself? Once you have the note in your hand, does your search strategy matter for deducing who the author is?

Most people’s intuition screams, "Of course not! The note is the note. Its handwriting, its ink, its message—that's the evidence. My search protocol is irrelevant." If your intuition aligns with this, you have just grasped the spirit of one of the most profound and controversial ideas in the philosophy of science: the ​​Likelihood Principle​​. It is a principle of beautiful simplicity, yet its consequences ripple through the foundations of statistics, drawing a bright line between two major schools of thought.

To formalize this intuition, we need a tool. That tool is the ​​likelihood function​​. It’s a bit of a backward-sounding idea at first, but it's the key to everything. Let's say we're flipping a coin, not to see what comes up, but to deduce the coin's intrinsic nature—its probability of landing heads, a parameter we can call $p$. If $p=0.5$, it's a fair coin; if $p=0.8$, it's heavily biased toward heads. The parameter $p$ is the "culprit" we are trying to identify.

Suppose we flip the coin 10 times and get a specific sequence: HTHHHTHTTT. The probability of this exact sequence, for a coin with a given bias $p$, is $p \times (1-p) \times p \times \dots = p^5 (1-p)^5$.

Now, let's turn this on its head. We have the data—the sequence of flips. What does it tell us about $p$? We can construct a function, $L(p; \text{data})$, which asks, for every possible value of $p$, "How probable was my observed data, assuming this value of $p$ is the truth?" This function is the likelihood function.

$L(p; \text{data}) = P(\text{data} | p)$

This is not the probability of $p$. It's the probability of the data, but we are viewing it as a function of the parameter $p$. It's like having a lineup of suspects (all the possible values of $p$ from 0 to 1) and asking each one, "If you were the true culprit, what's the probability you would have left this specific clue?" The suspect for whom the clue is most probable has the highest "likelihood." The shape of this function over all possible values of $p$ acts like a character profile, telling us which values of the parameter are plausible and which are not, in light of the evidence.

This brings us back to our detective's dilemma, now cast in a scientific light. Consider two scientists, both trying to estimate the prevalence of a certain genetic mutation, which has an unknown probability $\theta$.

• ​​Scientist A​​ decides on a ​​fixed-sample design​​. She will collect exactly $n=20$ tissue samples and count how many have the mutation. She runs her experiment and observes $s=12$ positive results.

• ​​Scientist B​​ prefers a ​​sequential design​​. She will keep collecting samples until she finds exactly $s=12$ positive results, and then stop. By coincidence, she also stops after collecting a total of $n=20$ samples.

Both scientists go to their computers with the exact same raw data: 12 positives and 8 negatives. They have the same note from nature. Should their conclusions about the prevalence $\theta$ be identical?

Let's look at their likelihood functions.

For Scientist A, the probability of getting $12$ successes in $20$ fixed trials is given by the Binomial distribution: $L_A(\theta; \text{data}) = P(\text{12 successes in 20 trials} | \theta) = \binom{20}{12} \theta^{12} (1-\theta)^{8}$

For Scientist B, the probability that the 12th success occurs on the 20th trial is given by the Negative Binomial distribution. This means there were 11 successes in the first 19 trials, and the 20th trial was a success. $L_B(\theta; \text{data}) = P(\text{12th success on 20th trial} | \theta) = \binom{19}{11} \theta^{12} (1-\theta)^{8}$

Now look closely at these two functions. They are not identical. The numbers at the front, the combinatorial coefficients, are different: $\binom{20}{12} = 125970$ while $\binom{19}{11} = 75582$.

But here is the crucial insight: with respect to the parameter of interest, $\theta$, the functions have the exact same form. Both are proportional to the term $\theta^{12}(1-\theta)^{8}$. This term is the ​​likelihood kernel​​, and it contains all the information about how the data's probability changes as we consider different values of $\theta$. One function is just a constant multiple of the other: $L_A(\theta) \approx 1.67 \times L_B(\theta)$. If you were to plot them, they would have the exact same shape, with one just scaled vertically compared to the other.

This leads us to the formal statement of the ​​Likelihood Principle​​:

All of the evidence about a parameter $\theta$ which is obtained from an experiment is contained in the likelihood function for $\theta$ given the data. Two experiments that yield proportional likelihood functions contain the same evidence and should lead to identical inferences about $\theta$.

According to this principle, Scientist A and Scientist B have the same evidence and must draw the same conclusion. The stopping rule—the private intention of the scientist that determined when the experiment would end—is irrelevant to the interpretation of the data once it's collected. The only thing that matters is what did happen, not what might have happened.

Why on earth would this be controversial? It seems so obvious. The controversy arises because a major branch of statistics, the ​​frequentist​​ or ​​sampling-based​​ approach, is built fundamentally on considering what might have happened.

For a frequentist, the reliability of a procedure is judged by its long-run performance over hypothetical repetitions of the experiment. To calculate a ​​p-value​​, for instance, one asks: "Assuming the null hypothesis is true (e.g., the coin is fair, $\theta=0.5$), what is the probability of observing data as extreme, or more extreme, than what I actually saw?"

Here's the catch: the set of "more extreme" outcomes depends entirely on the stopping rule!

• For Scientist A (fixed $n=20$), "more extreme" means getting 13, 14, ..., up to 20 heads. Her sample space of possibilities is finite. The p-value for her result (12 heads out of 20, testing against $\theta > 0.5$) is about $0.252$.

• For Scientist B (fixed $s=12$), "more extreme" means needing 19, 18, or fewer flips to get her 12 heads. Her sample space is different. The p-value for her result (getting 12 heads in 20 flips) is about $0.180$.

Think about that! With the exact same data, Scientist A concludes her result is not statistically significant ($p > 0.05$), while Scientist B finds her result is bordering on significance. A frequentist is forced to conclude that the scientists' intentions change the meaning of the data. This is a direct violation of the Likelihood Principle.

This isn't just a quirk of coin flips. It applies everywhere.

• In a high-energy physics experiment at the LHC, one team might run their detector for a fixed amount of time (analogous to Scientist A), while another team runs it until they see a certain number of interesting particle events (analogous to Scientist B). If they happen to stop with the same data, the Likelihood Principle says their inference about the underlying physics should be identical, but frequentist confidence intervals would differ.
• In survey sampling, one team might draw a simple random sample of 200 patients (fixed size), while another uses a method where each patient in the hospital has a 2% chance of being included, which happens to result in a sample of 200. Again, the data is the same, but the design-based (frequentist) estimates of variance would be different, because they depend on the sampling process.

There is another way. ​​Bayesian inference​​ provides a framework that naturally and automatically respects the Likelihood Principle. Bayes' theorem tells us how to update our beliefs in light of evidence:

$\text{Posterior belief} \propto \text{Likelihood} \times \text{Prior belief}$ $p(\theta | \text{data}) \propto L(\theta; \text{data}) \times p(\theta)$

Look at the formula! The data enters only through the likelihood function. If Scientist A and Scientist B start with the same prior beliefs about $\theta$, and their likelihood functions $L_A(\theta)$ and $L_B(\theta)$ are proportional, then their posterior beliefs $p_A(\theta | \text{data})$ and $p_B(\theta | \text{data})$ must be identical. The different combinatorial coefficients from their stopping rules are just constants that get absorbed into the final normalization of the posterior distribution. The stopping rule becomes irrelevant, just as our intuition demanded.

This is a beautiful result. It shows that the Bayesian framework provides a unified way of understanding evidence. The evidence is what it is, and its meaning doesn't depend on the private thoughts or intentions of the experimenter. Even in more complex scenarios, like a medical trial where the number of patients in the treatment and control groups is random, the same logic holds. Conditioning on facts that don't depend on the parameter of interest (like the final group sizes) leads to proportional likelihoods and shouldn't change our inference, yet it can and does change the results of frequentist tests.

This principle is not just a philosophical curiosity. It is the logical consequence of accepting even more basic ideas. A famous result by Allan Birnbaum in 1962 showed that if you accept two seemingly self-evident principles—the ​​Sufficiency Principle​​ (that you can summarize data with a sufficient statistic without losing information) and the ​​Conditionality Principle​​ (that you should only consider the experiment you actually performed, not other experiments you could have performed—then you are logically forced to accept the Likelihood Principle.

The Likelihood Principle, therefore, stands as a central guidepost in the theory of evidence. It challenges us to think deeply about what it means to learn from data. It suggests that the most direct and purest representation of what the data has to say is captured by the elegant shape of the likelihood function, a profile of plausibility for every possibility we seek to understand.

In our last discussion, we explored the mechanics of the Likelihood Principle. It seemed, perhaps, like a rather abstract and formal rule of statistical logic. Now, we are going to see what happens when this principle leaves the pristine world of theory and collides with the messy, unpredictable business of collecting real data. We will find that this simple idea has profound, beautiful, and sometimes startling consequences. It forces us to ask a very deep question: What, precisely, is scientific evidence? Is it only the data we have in hand, or does it also include the data we might have collected but didn't?

Consider a modern clinical trial for a new cancer drug. Researchers are desperate for an answer, and patients' lives are on the line. It is incredibly tempting to "peek" at the data as it accumulates. Suppose after 100 patients, the results look promising. Should we stop the trial early and give the drug to everyone?

From a frequentist perspective, every peek is a new hypothesis test. If you test with a significance level of $\alpha = 0.05$ (a 1 in 20 chance of a false positive), and you peek at the data 10 times, your chance of getting at least one false positive is not $0.05$ anymore; it's much higher! For independent tests, it would be $1 - (1-0.05)^{10} \approx 0.40$. You've inflated your error rate by "data peeking". To maintain their standard of long-run error control, frequentists have to invent complex adjustments, essentially making it much harder to claim a discovery at any given peek.

The Bayesian, guided by the Likelihood Principle, sees no such problem. The evidence is simply the cumulative data. Peeking is just updating your beliefs with new information. The posterior distribution after 100 patients simply becomes the prior distribution for the next batch of patients. No adjustments are needed, because the framework isn't concerned with hypothetical error rates over experiments that never happened. It is concerned only with the degree of belief justified by the data observed so far.

This same battle is being fought today on the frontiers of artificial intelligence. When a hospital validates a new AI model for detecting sepsis, how should it collect evidence? Should it use a fixed set of 120 patient cases (like Dr. Fixed), or sequentially gather cases until it has found 84 correct detections (like Dr. Sequential)? The Likelihood Principle tells us that if the final data tally is the same, the evidence for the AI's sensitivity is the same. This suggests that methods of reporting evidence that adhere to the principle, such as likelihood ratios or Bayesian posterior probabilities, are more direct measures of the evidence itself. In this view, the familiar $p$-value is not so much a measure of evidence from a particular experiment, but a calibration tool for controlling errors in the long run.

The journey of the Likelihood Principle takes us to a profound destination. It champions a vision of scientific inference built on evidential coherence. It argues that the strength of evidence in your data should not depend on your private intentions or on outcomes that never came to pass. The data should be allowed to speak for itself.

While this view remains in a healthy and productive tension with the frequentist philosophy of long-run error control, understanding it opens our eyes. It reveals an underlying unity and elegance in the way we can learn from the world, a direct path from observation to belief, guided by the simple, powerful, and beautiful logic of the Likelihood Principle.