Objective Priors

One of the most powerful and debated aspects of Bayesian statistics is the choice of a prior distribution. While subjective priors allow us to encode expert knowledge, a fundamental question arises: how should we proceed when we lack such information or wish to conduct a purely data-driven analysis? Simply claiming "ignorance" with a uniform prior can paradoxically introduce unintended biases, as the meaning of "uninformative" changes depending on how a problem is parameterized. This article addresses this knowledge gap by exploring the quest for ​​objective priors​​—formal, principled methods for constructing priors that are impartial and reproducible.

This exploration will guide you through the sophisticated reasoning that allows statisticians to "let the data speak for itself." In the "Principles and Mechanisms" chapter, we will uncover the illusion of a truly blank slate, discover the guiding principle of invariance, and delve into the mechanics of the Jeffreys prior and the more advanced reference prior framework. Subsequently, the "Applications and Interdisciplinary Connections" chapter will reveal the practical power of these concepts, demonstrating how they unify classical and Bayesian statistics and solve complex problems in fields ranging from fundamental physics to modern cosmology.

Having opened the door to the Bayesian world, we now confront its most debated, and perhaps most fascinating, aspect: the prior. When we have a wealth of experience or expert knowledge, we can pour it into a subjective prior distribution. But what do we do when we are explorers in a new land, with no map to guide us? What do we write down when we feel we know nothing? This is the quest for ​​objective priors​​: a set of principles for crafting a starting point for inference that is impartial, reproducible, and lets the data tell its own story.

The simplest answer to "what do you believe beforehand?" might seem to be "nothing in particular." If we are estimating the bias of a coin, represented by the probability of heads $p$, it feels natural to assume that any value of $p$ between $0$ and $1$ is equally likely. This is the ​​uniform prior​​, $\pi(p) = 1$. It feels like a blank slate, a perfect expression of ignorance.

But this elegant simplicity is a mirage. Suppose a physicist is not interested in the probability $p$, but in a related parameter, say $\theta = \sin^{-1}(\sqrt{p})$, which might arise in a quantum mechanical model. Or perhaps a gambler is interested in the odds, $o = p/(1-p)$. If we are truly ignorant about $p$, shouldn't we also be ignorant about $\theta$ and $o$?

Let's see. A flat prior on $p$ is not flat for other parameters. For the odds $o$, the uniform prior on $p$ transforms into $\pi(o) \propto 1/(1+o)^2$, a distribution heavily skewed towards small odds. Our "ignorance" has suddenly developed a strong opinion! This is precisely the issue explored in the context of measuring an unknown physical variance, $\sigma^2$. A flat prior on the standard deviation, $\pi(\sigma) \propto 1$, seems as reasonable as a flat prior on the variance, $\pi(\sigma^2) \propto 1$. Yet, they are not the same; the former corresponds to a prior on the variance $\sigma^2$ that is proportional to $(\sigma^2)^{-1/2}$. As demonstrated in a classic statistical puzzle, these two seemingly innocent choices for a "non-informative" prior lead to measurably different conclusions from the same data.

The lesson is profound: there is no universal language of ignorance. A statement of "equal likelihood" depends entirely on the parameterization you choose to describe the problem. The idea of a truly blank slate is an illusion.

If we cannot rely on the parameter's label, what can we rely on? The brilliant insight, pioneered by the geophysicist and mathematician Sir Harold Jeffreys, was to let the statistical model itself define the prior. We need a principle that is ​​invariant​​ to the way we write down our parameters. The prior should reflect the intrinsic structure of the problem, not the arbitrary labels we assign.

Jeffreys' solution was to use a concept from information theory called ​​Fisher information​​, denoted $I(\theta)$. Imagine the parameter $\theta$ controls the shape of a probability distribution from which we draw our data. The Fisher information $I(\theta)$ measures how much a tiny nudge to the parameter $\theta$ changes the shape of that distribution. If $I(\theta)$ is large, even a small change in $\theta$ creates a very different, and thus statistically distinguishable, distribution for the data. If $I(\theta)$ is small, the distribution is insensitive to changes in $\theta$, and it is hard to tell nearby values apart.

In a sense, Fisher information defines a kind of "statistical geometry" on the space of possible parameters. The distance between two parameters is not their numerical difference, but how easy it is to tell them apart using data.

​​Jeffreys' rule​​ is then stunningly elegant: the prior probability for $\theta$ should be proportional to the square root of the Fisher information.

$\pi(\theta) \propto \sqrt{I(\theta)}$

This prior effectively spreads our belief evenly across the "terrain" of the parameter space, where the terrain is defined by statistical distinguishability. The magic is that this geometric definition is invariant. It doesn't matter if you parameterize by probability $p$, odds $o$, or variance $\sigma^2$; the underlying geometry is the same, and the Jeffreys prior gives a consistent answer.

For instance, for a power-law model, $f(x;\alpha) \propto x^{-\alpha}$, which describes phenomena from earthquake magnitudes to city populations, Jeffreys' rule unambiguously gives the prior for the exponent $\alpha$ as $\pi(\alpha) \propto 1/(\alpha-1)$. It is derived, not arbitrarily chosen.

How does this principled approach compare to the naive uniform prior? Let's return to our coin-flipping experiment. For a binomial process, the Jeffreys prior is $\pi_J(p) \propto p^{-1/2}(1-p)^{-1/2}$. This is not a flat line; it is a U-shaped curve, placing more prior belief near $p=0$ and $p=1$. This might seem counterintuitive, but it reflects a deep truth: it is much easier to distinguish a coin with $p=0.5$ from one with $p=0.6$ than it is to distinguish a coin with $p=0.99$ from one with $p=0.999$. The Jeffreys prior accounts for this difference in statistical sensitivity.

As a consequence, the posterior estimates derived from a Jeffreys prior and a uniform prior will, in general, be different. For a binomial experiment with $k$ successes in $n$ trials, the posterior mean for $p$ under a uniform prior is $\frac{k+1}{n+2}$, while under the Jeffreys prior it is $\frac{k+1/2}{n+1}$. These values are close, but not identical. The choice of objective prior matters. Interestingly, a hypothetical experiment reveals that these two estimates coincide only when the data is perfectly balanced, with $k = n/2$. In this moment of perfect symmetry, the different perspectives of the two priors lead to the same conclusion.

Jeffreys' prior is a monumental step forward, but the story doesn't end there. Its application reveals further subtleties and challenges, pushing us toward an even deeper understanding of objectivity.

One immediate feature of many Jeffreys priors is that they are ​​improper​​. This means they do not integrate to a finite value; for example, a flat line over the entire real number line. They are not, strictly speaking, probability distributions. This sounds like a fatal flaw, but in a beautiful piece of mathematical pragmatism, it often isn't. An improper prior can be thought of as a scaffolding. As long as the data (the likelihood) is informative enough to combine with the prior to produce a posterior distribution that is proper (i.e., integrates to 1), the inference is perfectly valid.

However, this requires caution. In some situations, particularly in complex hierarchical models, the data may not be sufficient to "tame" the improper prior. One can end up with an improper posterior, which is mathematical nonsense. A fascinating case arises in random effects models, where using standard improper priors only yields a valid posterior if the data meets certain minimum requirements—for example, that at least one of the groups being studied contains two or more measurements. Without this minimal data structure, the entire Bayesian calculation collapses.

A second, more subtle issue arises when dealing with multiple unknown parameters. For a normal distribution with unknown mean $\mu$ and standard deviation $\sigma$, the standard multivariate Jeffreys' rule gives a joint prior $\pi_J(\mu, \sigma) \propto 1/\sigma^2$. However, other arguments based on group invariance suggest a prior of $\pi(\mu, \sigma) \propto 1/\sigma$. This latter prior is also what emerges from the more advanced "reference prior" framework we will discuss shortly. Though the numerical differences in the final estimates for the variance are often small—differing by a factor of $\frac{n-3}{n-2}$, where $n$ is the sample size—the discrepancy raises a flag. Why should our "objective" rule give a different answer from other principled approaches?

This is not to say the Jeffreys prior is a failure. It provides a robust and elegant solution to many famously difficult problems, such as the Behrens-Fisher problem of comparing the means of two normal populations with unknown and unequal variances. But the tensions in multiparameter models, including strange inconsistencies known as ​​marginalization paradoxes​​ where different setups lead to contradictory posteriors for the same quantity, suggest that a still more refined principle is needed.

The culmination of this search is the theory of ​​reference priors​​, developed primarily by José-Miguel Bernardo and James Berger. The philosophy behind reference priors is both pragmatic and profound. It posits that a prior is "objective" if it is designed to maximize the information that the data can provide. It is a prior that, in a formal sense, lets the data speak for itself as loudly as possible.

The crucial insight is that the form of this maximally non-informative prior depends on which parameter you are most interested in. Objectivity becomes relative to the scientific question being asked.

Consider the Pareto distribution, used to model phenomena like wealth inequality, which depends on a minimum value $x_m$ and a shape parameter $\alpha$. Suppose we have two parameters to estimate, but our primary focus is on just one. The reference prior algorithm tailors the prior to that specific goal.

• If an economist's primary interest is in the degree of inequality, which is governed by the ​​shape parameter $\alpha$​​, the reference prior is $\pi_1(\alpha, x_m) \propto \frac{1}{\alpha x_m}$.
• However, if a policymaker is more concerned with the poverty line, represented by the ​​minimum value $x_m$​​, the reference prior for this different question is $\pi_2(\alpha, x_m) \propto \frac{1}{x_m}$.

The prior changes because the question has changed! This is not a failure of objectivity, but its refinement. The reference prior provides a standardized, formal procedure for deriving a prior that is minimally informative with respect to a specific parameter of interest. This tailored approach is constructed precisely to avoid the paradoxes that can affect the one-size-fits-all Jeffreys' rule in multiparameter settings.

The journey from a simple uniform prior to the sophisticated machinery of reference priors is a testament to the depth and beauty of statistical thinking. It reveals that objectivity is not a passive state of ignorance, but an active process of principled reasoning. These methods provide a bedrock for Bayesian inference, allowing scientists to generate reproducible results and, importantly, to derive posterior distributions that yield direct probabilistic statements about the world. They allow us to compute a 95% ​​credible interval​​ and say, "Given our model and data, there is a 95% probability the true value lies in this range"—an intuitive and powerful conclusion that is often the very goal of scientific inquiry.

What is the point of all this abstract machinery? We have talked about principles of invariance and information, but the real test of any scientific tool is what it allows us to do. Does it help us see the world more clearly? Does it solve problems that were once intractable? The beauty of objective priors lies not just in their theoretical elegance, but in their extraordinary range of application, connecting disparate fields and revealing a surprising unity in our methods of scientific discovery.

Imagine you are an impartial judge, tasked with weighing evidence. You want to begin with no preconceived biases, to let the facts speak for themselves as loudly as possible. This is the role an objective prior plays in a Bayesian analysis. It's not a statement of "true" belief, but a baseline, a reference point from which any rational observer should start. It’s the mathematical embodiment of an open mind.

One of the most startling and beautiful discoveries one makes with objective priors is that they do not discard the centuries of statistical wisdom that came before them. Instead, they provide a deeper, more unified foundation for it.

Consider one of the most fundamental tasks in all of science: measuring a quantity. We take a series of readings, which have some average value and some scatter. We assume the readings come from a normal distribution, the familiar bell curve, but we don't know its true mean $\mu$ or its variance $\sigma^2$. If we apply the standard Jeffreys' prior, which formalizes our ignorance about these parameters, and then ask "What do we now know about the true mean $\mu$?", the Bayesian machinery gives us a remarkable answer. The posterior distribution for a particular standardized quantity involving $\mu$ is none other than the famous Student's t-distribution.

This is not a mere coincidence. The t-test, a cornerstone of frequentist statistics for over a century, appears here as a direct consequence of a Bayesian analysis starting from a state of professed ignorance. It tells us that the classical methods, developed through entirely different reasoning, were hitting upon a profound truth. The objective Bayesian framework reveals why these methods work, grounding them in the logic of probability theory itself.

This unifying power extends further. Are two manufacturing processes equally precise? We can compare the variance in their outputs. A Bayesian analysis using non-informative priors on the unknown variances of two production lines reveals that the posterior distribution for their ratio, $\phi = \sigma_1^2 / \sigma_2^2$, is directly related to the F-distribution. Similarly, when we analyze a simple linear relationship between two variables—the very heart of regression analysis—the uncertainty in our estimated slope, when viewed through the lens of objective priors, is again captured by an F-distribution. In each case, a celebrated tool from the classical statistics toolkit (the t-test, the F-test) is reborn, not just as a procedure, but as a logical deduction about our state of knowledge.

This framework is far more than a way to re-derive old results. It is a powerful, practical tool for scientific measurement. Let's say we are physicists trying to measure the drag on an object moving through a fluid. The theory gives us a simple linear relationship between the reciprocal of velocity and time. We take measurements, but they are noisy. How do we best estimate the drag parameter, $\beta$?

If we set up a Bayesian model with standard, non-informative priors for $\beta$ and the unknown noise level, the result for the posterior mean of $\beta$ is precisely the same as the value obtained from the classical method of least squares. But the Bayesian approach gives us so much more. It doesn't just give a single "best" estimate; it gives us a full probability distribution for $\beta$, a complete characterization of our uncertainty. We can ask "What is the probability that $\beta$ is greater than some critical threshold?" and get a direct, meaningful answer.

Let's raise the stakes. Instead of a simple drag coefficient, suppose we are trying to measure a fundamental constant of nature, like the vacuum permeability, $\mu_0$, which governs the strength of magnetic forces. An experiment based on Ampere's force law between two current-carrying wires gives us a set of noisy measurements. Once again, we can apply the same Bayesian machinery with objective priors. And once again, the most probable value for our parameter of interest turns out to be exactly what the time-tested method of least squares would have suggested. The same logical framework that helps a quality control engineer compare production lines also allows a physicist to make a statement about the very fabric of the universe.

This power scales to the frontiers of modern science. Cosmologists trying to understand the instant after the Big Bang search for faint signatures of "primordial non-Gaussianity" in the Cosmic Microwave Background (CMB), the afterglow of creation. This signature is parameterized by a number, $f_{\text{NL}}$. The analysis is fiendishly complex. The data is noisy, and the model contains other "nuisance" parameters that we don't care about, like instrumental offsets and noise levels. The great challenge is to separate the wheat (the signal of $f_{\text{NL}}$) from the chaff (the nuisance parameters and the noise). Objective priors provide a breathtakingly elegant solution. By assigning non-informative priors to the nuisance parameters, we can mathematically "average over" our ignorance of them, a process called marginalization. This isolates the posterior probability for the one parameter we truly care about, $f_{\text{NL}}$. The result, yet again, connects beautifully to classical ideas, yielding a form of weighted least-squares estimate, but now derived from a fully probabilistic framework capable of handling the immense complexity of modern cosmological data analysis.

So far, our examples have been well-behaved, mostly relying on the friendly bell curve. But what happens when nature doesn't play by such clean rules? What if our data is contaminated by wild outliers?

Consider data drawn from a Cauchy distribution. This is a statistician's nightmare. It has such heavy tails that its mean and variance are infinite. Taking more and more measurements does not help the sample average converge to anything. Classical methods that rely on means and variances break down completely.

Does the Bayesian approach also fail? Not at all. Let's imagine we have just two data points, $y_1$ and $y_2$, from a Cauchy distribution whose location $\mu$ and scale $\gamma$ are unknown. We apply the standard objective priors for these parameters. Even in this pathological case, the wheels of Bayesian inference turn smoothly. We can marginalize out the unknown location $\mu$ to find the posterior distribution for the scale parameter $\gamma$. The result is not only well-defined but strikingly simple and intuitive: the posterior median for the scale parameter turns out to be simply half the distance between the two data points, $\frac{1}{2}|y_1 - y_2|$. This demonstrates the incredible robustness of the framework. It doesn't crash when faced with unruly data; it gracefully yields a sensible answer that reflects the information actually present in the observations.

Our journey has taken us from the foundations of statistics to the edge of the cosmos. We have seen how a single, coherent set of principles can unify old ideas, empower new discoveries, and remain robust in the face of the unexpected. The philosophy of objective priors is a quest to make our statistical inferences as independent as possible from the arbitrary choices of the observer.

In the end, it is a story about invariance. A physical law should not depend on the units we use to measure it, or the coordinate system we choose to describe it. In the same way, a statement of evidence should not depend on the arbitrary mathematical parameterization we happen to choose for our model. Objective priors are those that respect these symmetries. They are the ones that change in just the right way when we change our description, so that the final physical conclusion remains the same. This is the heart of objectivity in science: to find a language to describe the world that is a property of the world itself, not merely a reflection of ourselves.