Approximate Bayesian Computation

In the quest to understand the world, Bayesian inference is a cornerstone, allowing scientists to update their beliefs about a model's parameters in light of new data. At the heart of this process lies the likelihood function—a mathematical expression that quantifies how probable our observed data is, given a specific set of model parameters. For simple systems, this function is easily defined. However, as our models grow to reflect the true complexity of nature, from the genetic history of a species to the evolution of the cosmos, the likelihood often becomes a mathematical monster, too complex to calculate. This "tyranny of the likelihood" presents a formidable barrier, seemingly halting Bayesian analysis for many of the most interesting scientific questions.

How, then, can we make inferences when the central component of Bayes' theorem is beyond our grasp? This article introduces Approximate Bayesian Computation (ABC), a revolutionary class of methods that elegantly sidesteps this problem. ABC represents a philosophical shift: if you cannot analytically calculate the probability of your data, but you can simulate the process that generates it, you can still perform inference. It is a powerful, simulation-based approach that has opened up new frontiers of discovery.

This article will guide you through the world of likelihood-free inference. In the first chapter, "Principles and Mechanisms," we will dissect the core logic of ABC, exploring how it turns a problem of intractable mathematics into a solvable challenge of computation through clever approximations. Following that, the "Applications and Interdisciplinary Connections" chapter will take you on a tour across scientific disciplines, showcasing how ABC is used as a powerful lens to reconstruct the past and understand the complex systems of the present.

The goal of a scientist is often to play detective. We have data—the "clues"—and a set of suspects, which are the different ways our scientific model could have produced those clues. In Bayesian inference, our "model" is a mathematical machine defined by a set of parameters, which we'll call $\theta$. These parameters are the knobs and dials on our machine—think the strength of gravity in a cosmological model, or a mutation rate in a genetic one. Our job is to use the observed data, let's call it $D_{obs}$, to figure out which values of $\theta$ are plausible and which are not. Bayes' theorem is our master key for this, elegantly stating that the plausibility of our parameters after seeing the data (the ​​posterior distribution​​, $p(\theta | D_{obs})$) is proportional to how plausible they were before we saw the data (the ​​prior distribution​​, $p(\theta)$) multiplied by a crucial term: the ​​likelihood​​, $p(D_{obs} | \theta)$.

The likelihood is the heart of the matter. It asks a simple question: "If the true parameters of the universe were $\theta$, what would be the probability of observing the exact data, $D_{obs}$, that we did?" For simple models, we can write down a nice, clean formula for the likelihood. But what happens when our models become as complex and messy as reality itself?

Imagine trying to write down the exact probability of a specific arrangement of millions of DNA letters across a population of birds, accounting for their migration patterns, historical population bottlenecks, natural selection, and the random shuffling of genes over thousands of generations. The formula for this likelihood would be a mathematical monster, an integral over a mind-bogglingly vast space of possible ancestral histories. It's what mathematicians call ​​intractable​​—a polite word for "impossible to calculate." For a huge class of problems at the frontiers of science, from epidemiology to astrophysics, we can write down the rules of our model, but we cannot write down its likelihood function. It seems we are stuck. How can we possibly perform Bayesian inference if the central piece of Bayes' theorem is beyond our grasp?

This is where a beautifully simple, almost philosophical, shift in perspective comes to the rescue. The idea is this: what if we don't need to calculate the likelihood at all? We have a model, a machine whose rules we understand perfectly. We may not be able to write down the formula for what it produces, but we can run it. We can simulate it. This is the core insight of ​​Approximate Bayesian Computation (ABC)​​.

Let's use an analogy. Suppose you are a judge at a baking competition. A contestant gives you a cake ($D_{obs}$), but you've lost the recipe ($\theta$). You can't "un-bake" the cake to figure out the recipe (this is our intractable likelihood). But the baker is still in the kitchen. You can ask them to bake new cakes ($D_{sim}$) using various trial recipes ($\theta^*$). Your strategy is simple:

1. Guess a recipe (draw a parameter $\theta^*$ from your prior beliefs about what makes a good cake).
2. Ask the baker to bake a cake using that recipe (simulate a dataset $D_{sim}$ from the model $p(D | \theta^*)$).
3. Compare the new cake to the original. If it's a very close match, you conclude the trial recipe was a good one and you keep it.
4. Repeat this process thousands or millions of times.

The collection of recipes you end up keeping forms an approximation of the posterior distribution. You have inferred the plausible recipes without ever writing down the physics and chemistry of baking. This is why ABC is often called a ​​likelihood-free​​ method. It replaces a difficult analytical calculation with a computational brute-force simulation.

For a scientific example, consider inferring the strength of natural selection on a gene in a population. We can create a computer simulation of a population of organisms that live, reproduce, and die according to a set of rules (the Wright-Fisher model). We can set the selection strength parameter, $s$, and watch how the frequency of a gene changes over generations. To do ABC, we would repeatedly guess a value for $s$, run the simulation, and see if the final genetic makeup of our simulated population looks like the one we observed in the wild.

Of course, there is a catch. The simple procedure described above has a fatal flaw. The probability of simulating a cake that is identical to the observed one, down to the last crumb, is practically zero. We would stand in the kitchen forever, rejecting every single cake. To make this idea practical, ABC relies on a trinity of clever approximations.

The Summary Statistic: Distilling Data to its Essence

Instead of comparing the entire, overwhelmingly complex datasets, we compare a handful of carefully chosen characteristics, or ​​summary statistics​​. We don't compare the cakes crumb-for-crumb; we compare their weight, their height, their sugar content. In genetics, instead of comparing entire genomes, we might compare statistics like the average number of genetic differences between individuals or the degree of genetic divergence between populations.

This is the first source of approximation. By summarizing the data, we are inevitably throwing some information away. The key is to choose statistics that capture the most relevant information for the parameters we care about. If a statistic captures all the relevant information, it is called ​​sufficient​​. With a sufficient statistic, we lose nothing. In the real world, finding a low-dimensional set of sufficient statistics for a complex model is exceedingly rare. Therefore, the quality of our ABC inference is fundamentally limited by the wisdom of our choice of summaries. If we try to infer a parameter that mainly affects the pattern of linkage between genes, but we only use a summary statistic that ignores linkage (like the Site Frequency Spectrum), our inference will be poor, no matter how much computing power we throw at it.

The Tolerance: Defining "Close Enough"

The second approximation is that we don't demand an exact match even for the summary statistics. We introduce a ​​distance metric​​, $\rho$, to measure how far apart the simulated summary, $s(D_{sim})$, is from the observed one, $s(D_{obs})$. Then we define a ​​tolerance​​, $\epsilon$, and we accept the parameter proposal if the distance is within this tolerance: $\rho(s(D_{sim}), s(D_{obs})) \le \epsilon$.

This tolerance $\epsilon$ is the dial that controls the trade-off between accuracy and speed. If $\epsilon$ is large, we accept many proposals, and the computation is fast, but our approximation to the posterior is crude. If we shrink $\epsilon$ towards zero, our approximation gets better and better. In the theoretical limit where $\epsilon = 0$, ABC gives the exact posterior conditional on our summary statistics. But as $\epsilon$ shrinks, our acceptance rate plummets, and the number of simulations required can become astronomical.

We can visualize the effect of this tolerance beautifully. In a simple case where the true likelihood is a Gaussian (a bell curve), performing ABC with a Gaussian-shaped acceptance rule and a tolerance $\epsilon$ is mathematically equivalent to doing exact Bayesian inference, but on a "blurred" likelihood. The variance of the ABC likelihood becomes the sum of the true variance and an extra term related to $\epsilon^2$. The tolerance literally smears the likelihood, and the size of the smear is under our control. The good news is that the error this introduces typically shrinks in proportion to $\epsilon^2$, meaning the approximation gets better quite quickly as we tighten our standards.

The Distance Metric: The Rules of Comparison

The third crucial choice is the distance metric $\rho$ itself. How should we measure "distance" between summaries? If our summary vector has multiple components—say, one statistic that varies between 0 and 1, and another that varies from 100 to 1,000,000—a simple Euclidean distance will be completely dominated by the latter. The algorithm would focus all its effort on matching the large, noisy statistic, while ignoring potentially more informative signals from the smaller one.

The choice of distance defines the geometry of our acceptance region and directly shapes our final posterior. A more sophisticated approach involves scaling each summary statistic by its standard deviation, or better yet, using a ​​Mahalanobis distance​​. This advanced metric automatically accounts for the different scales of the statistics and also for any correlations between them. It's like putting on a pair of prescription glasses that allows the algorithm to properly weight the evidence from each piece of information, leading to a more efficient and accurate result.

Armed with these principles, ABC becomes a powerful but delicate dance of trade-offs. One of the biggest challenges is the ​​curse of dimensionality​​. It's tempting to think that adding more and more summary statistics will always improve our inference by bringing us closer to sufficiency. But each new statistic adds another dimension to the space in which we are measuring distance. The volume of high-dimensional space is notoriously counter-intuitive; the "acceptance region" (a hypersphere of radius $\epsilon$) becomes an infinitesimally small fraction of the total volume as the number of dimensions grows. This means our acceptance rate collapses, and the computational cost explodes. The art of ABC lies in choosing a small number of highly informative statistics.

To battle the immense computational cost, researchers have developed clever strategies. For example, ​​sequential ABC​​ uses a multi-stage filtering process. It first uses a cheap-to-compute, coarse summary to quickly reject the most outlandish parameter proposals, only proceeding to the expensive, full simulation and comparison for the more promising candidates. This can dramatically improve efficiency. Other techniques embed the ABC approximation within more sophisticated sampling algorithms like Markov chain Monte Carlo (MCMC), creating powerful hybrid methods.

Despite all these approximations, ABC rests on a solid theoretical foundation. One of its most powerful properties is ​​consistency​​. Even if our summary statistics are not sufficient for a finite amount of data, if they are chosen such that they converge to a unique value for each possible parameter as the amount of data grows to infinity, then the ABC posterior will concentrate on the true parameter value. This gives us confidence that, for the massive datasets of modern science, ABC is guiding us in the right direction. By allowing us to fit models that were previously out of reach, Approximate Bayesian Computation has opened up entirely new avenues for discovery, turning problems of intractable mathematics into solvable challenges of simulation and computation. It is a beautiful example of human ingenuity in the face of nature's complexity.

Having grasped the elegant mechanics of Approximate Bayesian Computation, we are now like explorers equipped with a new, powerful lens. With this lens, we can peer into the workings of systems so complex that their mathematical descriptions are utterly unwieldy. The true beauty of ABC lies not in its clever algorithm, but in its breathtaking universality. It is a tool for any discipline where we can tell a story—a generative model—of how something might have come to be, but cannot easily work that story backward from observation to cause. Let us embark on a journey across scientific domains to witness this toolkit in action.

Some of the most profound scientific questions are historical. We have but one present, a single snapshot of the cosmos, of life's diversity, of our own genetic heritage. Yet, from this single frame, we wish to reconstruct the epic movie of the past. This is where ABC shines as a veritable time machine. The logic is simple and profound: we test different historical "scripts" (models) by simulating them forward in time. The script that produces a simulated present most closely resembling our own is the one we favor.

Imagine trying to piece together the grand narrative of human migration. Genetic data from a newly found population offers clues, but how do we interpret them? Perhaps this group branched off from a single, large ancestral population in Africa (a "Single Source" model). Or perhaps it was formed from the mixing of two ancient, long-separated African groups (an "Admixed Source" model). We cannot write a simple equation for the probability of seeing their genomes given each story. But we can simulate both stories. We can simulate the genetic drift, mutation, and mixing under each scenario, millions of times over. By comparing the genetic summary statistics from our simulations to the real data, ABC allows us to calculate the posterior probability of each model. We can then weigh the evidence, even accounting for prior archaeological knowledge, to decide which story of our origins is more plausible.

This same logic scales from the history of our species to the history of all species. Consider two populations of pika, adorable mammals now isolated on separate mountain ranges. Did they arise when a single large population was split by a geological event like a glacier (vicariance)? Or did they split and continue to exchange a few migrants over the millennia (isolation-with-migration)? Or did one population arise recently from a few adventurous founders from the other (recent expansion)? Each is a distinct historical narrative. By simulating the genetic consequences of each story, ABC can calculate the Bayes Factor, a quantitative measure of how much more the observed genetic data supports one story over another. It can even help unravel the fascinating dynamics of "ring species," where a chain of populations encircling a barrier leads to the emergence of a new species, allowing us to test hypotheses about how the ring first formed.

The reach of this "time machine" extends from the biological to the cosmological. One of the ultimate historical questions is determining the fundamental parameters of our universe, like the total matter density, $\Omega_{\text{m}}$, and the clumpiness of cosmic structure, $\sigma_8$. Our model of the universe is no simple formula; it is a colossal computer simulation that evolves a virtual cosmos from the Big Bang to the present day. We cannot run this simulation backward. But we can run it forward many times with different settings for $\Omega_{\text{m}}$ and $\sigma_8$. We can then have our simulated universes "observed" by a virtual telescope to produce, for example, a histogram of weak gravitational lensing peaks—a measure of mass concentration. ABC compares these simulated histograms to the one from our real sky. By finding which parameter values generate universes that look like ours, we infer the fundamental constants of nature. In this grand arena, ABC helps us read the universe's own origin story.

Beyond reconstructing the past, ABC is an indispensable tool for dissecting the complex machinery of the present. From the intricate web of interactions inside a single cell to the emergent properties of entire ecosystems, nature is filled with systems whose behavior we can simulate but not analytically solve.

Consider the networks that form the backbone of life, like the web of protein-protein interactions in a cell. These networks often grow by "preferential attachment," where new nodes are more likely to connect to already popular nodes. This process, described by models like the Barabási-Albert model, has a key parameter, $m$, the number of links each new node makes. Given a final, static network, how can we infer the growth rule, the value of $m$, that created it? The full likelihood of observing a specific network topology is monstrously complex. Yet, we can easily simulate the growth process for different values of $m$. ABC allows us to infer $m$ by generating many simulated networks and finding which value of $m$ produces networks with a summary structure (like the Gini coefficient of the degree distribution) that matches the real one.

The lens of ABC can zoom in even further, to the stochastic dance of genes within a single cell. Imagine a synthetic gene circuit designed to have a positive feedback loop. We observe that a population of cells containing this circuit shows a bimodal distribution of fluorescence—some cells are "off" (low fluorescence) and some are "on" (high). Does this bimodality arise from true multistability, where individual cells can stochastically flip-flop between the on and off states? Or does it arise from extrinsic noise, where each cell is stably unimodal but cell-to-cell variations in cellular machinery create a bimodal population average? Both models can produce a similar static snapshot. The key is in the dynamics. A brilliant application of ABC, armed with summary statistics from time-lapse microscopy, can distinguish between these scenarios. By including summary statistics that capture the dynamics—like the fraction of cells that actually switch states over time—ABC can discern the true underlying mechanism, a feat impossible with snapshot data alone.

This power to infer the parameters of hidden processes extends to phenomena that bridge generations, like epigenetic inheritance. An organism's traits can be influenced by epigenetic marks, which themselves can be induced by the environment (like drought) and transmitted—imperfectly—to offspring. We can build a model with a latent, or hidden, state for the epigenetic mark. This model has parameters for environmental induction, $\alpha$, and for generational resetting, $r$. We can't see the mark directly, only a quantitative trait it affects. By tracking this trait over generations in a fluctuating environment, we can use ABC with carefully chosen summary statistics—some capturing the average response to the environment (for $\alpha$) and others capturing the temporal autocorrelation of the trait (for $r$)—to disentangle and estimate these fundamental parameters of non-genetic inheritance.

The power of ABC is so general that it connects to, and enhances, other advanced computational methods. Consider the problem of tracking a moving object in real-time based on noisy sensor data—a classic problem solved by "particle filters." A particle filter works by maintaining a cloud of "hypotheses" (the particles) about the object's current state. At each time step, it updates the plausibility of each hypothesis based on the new sensor data. This update requires a likelihood function. What if the likelihood is intractable? We can plug ABC right into the update step. For each hypothetical state, we simulate a "pseudo-observation" and assign a weight based on how close it is to the real observation. This creates an "ABC Particle Filter," a powerful hybrid algorithm for tracking hidden states in systems with intractable likelihoods, showcasing the incredible modularity of the ABC concept.

At this point, one might wonder if ABC is some kind of statistical magic. How can simply simulating and comparing give us a valid answer? The magic is demystified when we look at a simple case, like flipping a coin to estimate its bias, $\theta$. Here, the textbook answer is known. It turns out that if we run ABC using a "sufficient statistic"—a summary that captures all the information the data holds about the parameter (in this case, the sample mean is sufficient)—the ABC posterior distribution converges to the exact, correct Bayesian posterior as our acceptance tolerance $\epsilon$ goes to zero.

This is the soul of the machine. ABC is not magic; it is an approximation to exact Bayesian inference. In the complex problems of cosmology or genetics, we rarely have perfectly sufficient statistics. But the art and science of ABC lie in choosing summary statistics that are "good enough"—that capture most of the relevant information. This fundamental insight—that the choice of the summary statistic is the choice of the question we ask the data—is what unifies all these diverse applications. From the flick of a coin to the birth of the cosmos, Approximate Bayesian Computation provides a single, intuitive, and powerful framework for learning from our world by simulating it.