Objective Prior

In the world of Bayesian inference, every conclusion begins with a prior belief. But what if we want to approach a problem with as little bias as possible, letting the evidence guide our understanding? This pursuit of impartiality leads to the concept of an ​​objective prior​​, a foundational tool for letting data speak for itself. The central challenge, however, is that what appears "uninformative" in one context can be highly influential in another, a problem of perspective known as parameterization. How can we define ignorance in a way that is consistent, principled, and universally applicable?

This article delves into the elegant solution to this statistical puzzle. We will explore the principles behind objective priors, focusing on the groundbreaking work of Sir Harold Jeffreys. You will learn how the geometry of statistical information itself, quantified by Fisher information, provides a recipe for constructing priors that are invariant to the way we label our parameters. In the following chapters, we will first uncover the "Principles and Mechanisms" behind the Jeffreys prior, its properties of invariance, and its application to common parameter types. We will then journey through its "Applications and Interdisciplinary Connections," witnessing how this single statistical principle provides a common language for objective reasoning in fields as diverse as engineering, data science, and astrophysics.

Imagine you're a detective arriving at a crime scene. You have no suspects, no preconceived notions. You want to let the evidence speak for itself as much as possible. In the world of science and statistics, this is the role of an ​​objective prior​​. When we use Bayesian inference to learn from data, we must start with a "prior belief" about the parameters we're trying to estimate. But what if we want our initial belief to be as neutral, as "ignorant," as possible, so that the final conclusion is shaped almost entirely by the data? This is the quest for objectivity, and it's far more subtle than it first appears.

You might think the most objective starting point is to treat all possibilities as equally likely. If you're estimating the probability $p$ of a coin landing heads, why not just assume a flat, uniform prior where every value of $p$ from 0 to 1 is equally plausible? This is known as Laplace's "Principle of Insufficient Reason." It sounds reasonable, but it hides a deep problem.

The problem is one of perspective, or what mathematicians call ​​parameterization​​. We can describe the coin's bias using the probability of success, $p$. But we could just as easily describe it using the odds of success, $\omega = p / (1-p)$. Or the log-odds, $\ln(\omega)$. If we place a uniform prior on $p$, the prior on the odds $\omega$ is not uniform. A choice that seems "uninformative" in one description becomes informative in another. So, which description is the "correct" one? There is no "correct" one. Our principle of ignorance should not depend on the language we use to describe the problem.

We need a principle that is consistent, no matter how we label our parameters. We need a rule that gives the same underlying answer whether we're talking about a radioactive decay rate $\lambda$ or its inverse, the mean lifetime $\tau = 1/\lambda$. The answer to this puzzle came from the brilliant geophysicist and statistician Sir Harold Jeffreys.

Jeffreys had a profound idea. He suggested that a non-informative prior shouldn't be based on the parameter's values themselves, but on the information the data can provide about the parameter. The guiding principle should be: a prior is non-informative if it doesn't favor parameter values for which the data provides more information.

To make this concrete, he used a tool from statistics called ​​Fisher information​​, denoted $\mathcal{I}(\theta)$. You can think of Fisher information as a measure of the "sensitivity" of your experiment. It quantifies how much information a single data point gives you about an unknown parameter $\theta$. If the Fisher information $\mathcal{I}(\theta)$ is large, it means the likelihood function is sharply curved, and the data can pinpoint the value of $\theta$ very precisely. If it's small, the likelihood is flat, and the data is less helpful. In a sense, Fisher information defines a kind of "distance" or "geometry" on the space of parameters.

Jeffreys proposed a universal recipe: the prior probability for a parameter $\theta$, which we'll call $\pi_J(\theta)$, should be proportional to the square root of the Fisher information.

Why the square root? It’s a mathematical detail, but it’s precisely what’s needed to achieve the magic of invariance.

The true beauty of the ​​Jeffreys prior​​ is its property of ​​reparameterization invariance​​. This means that if you follow Jeffreys' rule, your conclusions will be consistent, regardless of the parameterization you choose.

Let's see this magic in action. Consider an engineer studying the failure of laser diodes, which follows an exponential distribution. She could model this with the failure rate, $\lambda$. Calculating the Fisher information, she finds that $\mathcal{I}(\lambda) = 1/\lambda^2$. The Jeffreys prior is then:

Now, suppose her colleague prefers to think in terms of the mean lifetime before failure, $\theta = 1/\lambda$. If he calculates the Jeffreys prior for his parameter $\theta$, he finds that the Fisher information is $\mathcal{I}(\theta) = 1/\theta^2$. So his prior is:

Look at that! The functional form is the same. A prior proportional to $1/\lambda$ for the rate is perfectly equivalent to a prior proportional to $1/\theta$ for the mean lifetime. One implies the other through the rules of probability transformation. There are no contradictions. The Jeffreys prior provides a self-consistent way to express ignorance.

This principle gives rise to some wonderfully simple and general rules for the two most common types of parameters we encounter in science.

Location Parameters: Where is it?

Imagine a parameter that simply shifts a distribution left or right without changing its shape. This is called a ​​location parameter​​. The mean $\mu$ of a Normal distribution (with known variance) is a classic example. Or perhaps the location $\mu$ of the peak of a Gumbel distribution used in climate science. The general form for the probability density is $f(x-\theta)$.

What does Jeffreys' rule say about our ignorance of such a parameter $\theta$? It tells us that the Fisher information $\mathcal{I}(\theta)$ is a constant. It doesn't depend on where the distribution is located. Therefore, the Jeffreys prior is also constant:

This is a flat, uniform prior over the entire real line. It says we have no reason to believe the center of the distribution is here rather than there. This might seem strange, because if you integrate a constant from $-\infty$ to $\infty$, you get infinity, not 1. Such a prior is called an ​​improper prior​​. But don't be alarmed! While it isn't a true probability distribution itself, it is a perfectly valid tool. Once we combine it with data to get a posterior distribution, the posterior is almost always a proper, well-behaved distribution.

Scale Parameters: How big is it?

Now consider a parameter that stretches or shrinks the distribution, like the standard deviation $\sigma$ of a Normal distribution or the mean lifetime $\theta$ of an exponential distribution. This is a ​​scale parameter​​. The general density has the form $\frac{1}{\sigma}f(x/\sigma)$.

For any such scale parameter $\sigma$, the Jeffreys prior is always the same:

This prior is also improper. Why this form? It formalizes the idea that our ignorance about scale should be on a logarithmic scale. A change in $\sigma$ from 1 to 2 should be just as significant as a change from 10 to 20, or from 100 to 200. It’s the percentage change that matters, not the absolute change. This prior assigns equal probability to all orders of magnitude. This rule applies to the standard deviation $\sigma$ of a normal distribution, and also to its variance $\sigma^2$. A quick calculation shows that if $\pi(\sigma) \propto 1/\sigma$, then the prior for the variance $\theta = \sigma^2$ is $\pi(\theta) \propto 1/\theta$, which is exactly what we find when calculating the Jeffreys prior for the variance of a Normal distribution directly. The consistency is beautiful.

Even for the simple problem of a coin flip, the Jeffreys prior gives a non-obvious result. For the success probability $p$, the prior is not uniform but is proportional to $p^{-1/2}(1-p)^{-1/2}$, which is a U-shaped Beta(1/2, 1/2) distribution. This prior says we should be more skeptical of values of $p$ near 0.5 and less surprised by values near 0 or 1 until we've seen some data. It reflects the fact that it's harder for data to distinguish between $p=0.5$ and $p=0.51$ than between $p=0.01$ and $p=0.02$.

The world of statistics, like physics, is full of beautiful theories that become more nuanced when you look closer. The Jeffreys prior is a triumph, but it's not the final word, especially when dealing with multiple unknown parameters at once.

Consider the classic case of a Normal distribution where both the mean $\mu$ (a location parameter) and the standard deviation $\sigma$ (a scale parameter) are unknown. Our intuition, based on the one-parameter rules, might be to simply multiply the individual priors: $\pi(\mu, \sigma) \propto \pi(\mu) \cdot \pi(\sigma) \propto 1 \cdot \frac{1}{\sigma} = \frac{1}{\sigma}$.

However, when we apply the formal multivariate Jeffreys' rule (which uses the determinant of the Fisher information matrix), we get a different answer:

This discrepancy has been the source of much debate. It turns out that the original Jeffreys' rule for multiple parameters can sometimes produce priors with undesirable properties. It's a reminder that even the most elegant principles can have surprising consequences.

This has led to further refinements, most notably the ​​reference prior​​ developed by José-Miguel Bernardo and James Berger. The reference prior is a more sophisticated algorithm that aims to maximize the expected information gain from the experiment. It often requires distinguishing between the "parameter of interest" and other "nuisance parameters". For the Normal distribution problem, if we declare that the mean $\mu$ is our primary interest and $\sigma$ is a nuisance, the reference prior algorithm gives:

This matches our original intuition! The quest for a truly "objective" prior is an ongoing story, a beautiful interplay of principles, mathematics, and philosophy. It shows how science progresses not by discarding old ideas, but by understanding their limitations and building upon their powerful foundations. Jeffreys' principle of invariance remains a central landmark, a guidepost in our journey to let the data speak as clearly as possible.

Now that we have tinkered with the machinery of the Jeffreys prior, learning its definition and its key property of invariance, a natural and pressing question arises: What is it good for? Is it merely a piece of elegant mathematical machinery, a curiosity for the theoretician? Or does it connect to the real world of scientific discovery and engineering practice?

The answer, you will be happy to hear, is a resounding "yes!" In this chapter, we will embark on a journey across disciplines to see this single, abstract principle in action. We will see how it provides a common thread, a universal language of objective reasoning, that ties together problems in engineering, data science, physics, and even the search for new worlds beyond our solar system. It is a beautiful illustration of how a deep mathematical idea can have an almost unreasonable effectiveness in the natural sciences.

Perhaps the most common type of parameter we encounter in science is a ​​scale parameter​​—a quantity that sets the characteristic size or duration of a phenomenon. Think of a half-life in radioactive decay, the average lifetime of a manufactured part, or the wavelength of a light wave. How do we express a state of "objective ignorance" about such a parameter?

Imagine you are a reliability engineer tasked with assessing a new microchip. The chip's lifetime is expected to follow an exponential distribution, governed by a failure rate parameter $\lambda$. A larger $\lambda$ means shorter lifetimes. You have no preconceived notions about this new technology. What is an objective prior for $\lambda$? The Jeffreys rule gives a clear answer: the prior should be proportional to $1/\lambda$.

This $\pi(\lambda) \propto 1/\lambda$ prior might seem strange at first, but it has a deep and intuitive logic. A scale parameter is something for which only relative magnitudes matter. A belief that $\lambda$ is between 1 and 2 should be just as strong as a belief that it is between 10 and 20, or between 100 and 200. In each case, the upper bound is twice the lower bound. This prior, when viewed on a logarithmic scale, is flat—it treats all orders of magnitude equally. It is a prior that is "ignorant" of the scale. This same prior form appears again and again. If we model the decay of a new radioactive isotope as having a maximum possible lifetime $\theta$, the Jeffreys prior for this upper limit is also $\pi(\theta) \propto 1/\theta$. The physical context is different—a rate versus a maximum value—but the underlying logic of scale invariance holds.

What is truly remarkable is what happens when we combine this prior with data. If we observe $N$ components and find their average lifetime is $\bar{x}$, the posterior mean for the failure rate $\lambda$ turns out to be exactly $1/\bar{x}$. This is the same answer a non-Bayesian statistician would arrive at using the method of maximum likelihood! In this case, the objective Bayesian approach formalizes and lands upon an answer that has long been known to be a good one.

Let's shift gears from continuous scale parameters to the world of counting and proportions. A data science team wants to estimate the true click-through rate, $p$, for a new feature on a website. Each user interaction is a success (click) or failure (no-click). Here, the parameter $p$ is a proportion, bounded between 0 and 1. The Jeffreys prior for this situation is a Beta distribution with parameters $\alpha=1/2$ and $\beta=1/2$, so $\pi(p) \propto p^{-1/2}(1-p)^{-1/2}$.

This prior has a peculiar U-shape, placing more weight on values of $p$ near 0 or 1. It is the prior's way of expressing maximal uncertainty: rather than guessing the proportion is near the middle (0.5), it acknowledges a significant possibility that the feature is either a complete dud or a runaway success. When combined with data—say, $k$ clicks out of $n$ sessions—the posterior mean becomes $\mathbb{E}[p|k,n] = \frac{k+1/2}{n+1}$. This is like starting with a "pseudo-observation" of half a success and half a failure, and then adding our actual data. It's a gentle nudge away from the extremes, a robust starting point for inference.

The power of this objectivity becomes clear when contrasted with a subjective approach. Imagine a senior scientist, pessimistic from past experience, sets a subjective prior that strongly favors a low success rate. If the early data is sparse (e.g., only 3 successes in 20 attempts), this pessimistic prior will heavily drag the final estimate downward. The Jeffreys prior, in contrast, provides a neutral ground, allowing the data—even if sparse—to have a greater say.

And this logic scales beautifully. If instead of two outcomes (click/no-click), we have $k$ possible categories—say, classifying galaxies into spirals, ellipticals, or irregulars—the Jeffreys rule generalizes. It yields a Dirichlet distribution where all parameters are $1/2$. This provides a consistent, objective foundation for multinomial problems across countless fields, from genetics (allele frequencies) to natural language processing (word frequencies).

The Jeffreys prior is more than just a collection of recipes for different problems. It is a single, unified principle rooted in the very geometry of statistical information. Consider a Poisson process, which models random events like the decay of radioactive nuclei or the arrival of photons at a telescope detector. The process is governed by a rate parameter $\lambda$. You might be tempted to think that since $\lambda$ is a "rate," its prior should be $1/\lambda$, just as in the exponential case.

But Nature is more subtle! The Jeffreys rule tells us the prior is actually $\pi(\lambda) \propto \lambda^{-1/2}$. Why the difference? Because the amount of ​​information​​ an observation gives us about $\lambda$ is different in the two models. The Fisher information is a metric, a way of measuring the "distance" between two slightly different probability distributions. The Jeffreys prior is proportional to the volume element of this space. The "informational geometry" of a Poisson process is fundamentally different from that of an exponential process, and the Jeffreys prior automatically and correctly reflects this.

This geometric viewpoint truly comes alive in complex, multi-parameter problems. Consider the search for exoplanets. A simplified transit model depends on two parameters: the planet-to-star radius ratio, $p$, and the transit's impact parameter, $b$. Applying the Jeffreys rule here does not yield a simple, flat prior. Instead, we get a complex function: $P(b,p) \propto \frac{p b}{\sqrt{(1+p)^2 - b^2}}$.

This result is magnificent. It shatters the naive idea that a "non-informative" prior must be uniform. This prior is anything but. It is warped by the geometry of the transit model itself. It inherently "knows" that certain combinations of $p$ and $b$ are harder to distinguish from the data than others, and it adjusts the prior weight accordingly. It is a map of the informational landscape of the problem, given to us for free by a universal principle.

The story of the Jeffreys prior has even more surprising chapters. It has a deep and unexpected connection to a completely different school of thought: frequentist decision theory and game theory. Imagine you are in a "game" against Nature. You must create an estimator for a proportion $p$. Nature will then choose a value of $p$ that makes your estimator look as bad as possible (maximizes your squared error). Your goal is to choose an estimator that minimizes this maximum possible error—a so-called "minimax" strategy. The stunning result is that, for a single observation, the prior that generates this minimax estimator is precisely the Jeffreys prior, $\text{Beta}(1/2, 1/2)$. It is as if two explorers, starting from different continents with different maps, arrived at the same hidden treasure. This suggests the Jeffreys prior is not merely a Bayesian convenience, but a fundamental object in the mathematical theory of inference.

However, this powerful tool is not a magic wand. An objective prior cannot create information out of thin air. In some cases, particularly with multiple parameters and very little data, using a Jeffreys prior can lead to a posterior distribution that is "improper"—it cannot be normalized to integrate to one, making probabilistic statements nonsensical. This is not a failure of the principle, but a profound warning from the mathematics: you cannot get something for nothing. Objectivity must be grounded by at least a small amount of empirical evidence to yield a coherent inference.

From the factory floor to the farthest stars, the Jeffreys prior provides a principled, unified framework for learning from data when we wish for that data to speak for itself as much as possible. It is a testament to the idea that in science, the deepest principles are often the ones that connect the most disparate-seeming phenomena, revealing a hidden unity in the logic of discovery.