Limit of Expectation

Is the long-term average of a random process the same as the average of its final outcome? This seemingly simple question—when can we swap the order of a limit and an expectation—lies at the heart of modern probability and its applications. While our intuition suggests the two should always be equal, this is a treacherous assumption that can lead to significant errors. This article addresses this critical knowledge gap by exploring why the interchange can fail and what mathematical tools are required to perform it safely.

In the chapters that follow, we will first delve into the "Principles and Mechanisms," using illustrative examples to understand the problem of "escaping mass" and introducing the powerful theorems designed to prevent it, such as the Monotone and Dominated Convergence Theorems. Subsequently, in "Applications and Interdisciplinary Connections," we will see how these abstract concepts are not mere technicalities but are the essential foundation for making predictions in fields ranging from finance and statistics to physics and engineering. This journey will reveal how a deep understanding of limits and expectations allows us to connect theoretical models to real-world phenomena.

Is the limit of an average the same as the average of a limit? In the language of mathematics, we are asking a simple, profound question: when does $\lim_{n \to \infty} E[X_n] = E[\lim_{n \to \infty} X_n]$ hold true? At first glance, it seems it should. If a sequence of random quantities $X_n$ is converging to some final random quantity $X$, shouldn't the average of $X_n$ also converge to the average of $X$? It feels like a matter of course. But in mathematics, as in life, what feels right is not always true.

Let's explore this with a thought experiment. Imagine a tiny "blip" of energy on a line segment from 0 to 1. For our first measurement, $X_1$, this blip has a height of 1 and is spread out over the interval $(0, 1]$. Its average value, which in this setting is its total energy, is $E[X_1] = \text{height} \times \text{width} = 1 \times 1 = 1$. For our second measurement, $X_2$, let's make the blip twice as tall but half as wide, so it has height 2 and covers the interval $(0, 1/2]$. Its average value is still $E[X_2] = 2 \times (1/2) = 1$. Let's generalize this. For the $n$-th measurement, define a random variable $X_n$ that takes the value $n$ on the tiny interval $(0, 1/n]$ and is 0 everywhere else.

Now, what happens as $n$ becomes enormous? Pick any specific point $\omega$ on the line (say, $\omega = 0.01$). For any $n > 100$, the interval $(0, 1/n]$ will be entirely to the left of your point. The blip has passed you by. Your measurement at that spot, $X_n(\omega)$, will be zero for all $n>100$ and will remain zero forever. This is true for any point you choose, as long as it's not exactly zero. So, the sequence of functions $X_n(\omega)$ converges to the function that is just $0$ everywhere. This limit function is our $X = \lim_{n \to \infty} X_n$. Its average is, of course, zero: $E[X] = E[0] = 0$.

But what about the limit of the averages? For every single $n$, we calculated that the average value is $E[X_n] = n \times (1/n) = 1$. The sequence of averages is simply $1, 1, 1, \dots$. The limit of this sequence is, undeniably, 1. So we have a shocking result:

$E\left[\lim_{n \to \infty} X_n\right] = 0 \quad \text{but} \quad \lim_{n \to \infty} E[X_n] = 1$

Our intuition has failed us spectacularly! The order in which we perform the operations of "limit" and "expectation" matters profoundly.

Why did our reasonable assumption fall apart? The problem is that while the blip's base gets narrower, its height grows in just the right proportion to keep its total area—its expectation—constant. The "probability mass," or in this case, the "expected value mass," doesn't vanish. It just consolidates into an infinitely tall, infinitesimally thin spike at the origin. It's a mathematical sleight of hand where value seems to disappear from every point but its total sum is magically preserved.

Consider another scenario, a strange kind of lottery. In week $n$, the grand prize is a staggering $n^2$ dollars, but your probability of winning is a measly $1/n$. Your probability of winning nothing is $1 - 1/n$. As weeks go by and $n$ soars, your chance of winning plummets towards zero. You can be almost certain that your outcome, the random variable $X_n$, will be $0$. The limit of your outcome is zero.

But what is your expected winning each week? It's calculated as $E[X_n] = (n^2) \times (1/n) + (0) \times (1 - 1/n) = n$. Your expected winning grows infinitely large! Once again, the limit of the expectation diverges to infinity, while the expectation of the limit is 0. A tiny, vanishing probability of a colossal outcome can keep the average high, even when the outcome is almost always zero.

In both these cases, some part of the value "escapes." In the blip example, it escapes into a singularity of infinite height. In the lottery example, it escapes to an infinitely large prize value. A similar effect can occur when a random variable is a mixture of different possibilities, and a small, vanishing fraction of the time it is drawn from a distribution whose range expands to infinity, carrying some of the expectation along with it. The core problem is this: for the limit and expectation to be interchangeable, we must ensure that no value can "escape" the system. We need to put a fence around it.

If we can't always swap limits and expectations, when can we? This is not a trivial question, and thankfully, mathematicians have given us powerful tools that act as safety guidelines.

The first, and simplest, is the ​​Monotone Convergence Theorem (MCT)​​. It says that if you have a sequence of non-negative random variables that are always increasing ($0 \le X_1(\omega) \le X_2(\omega) \le \dots$ for all outcomes $\omega$), then you are safe. The limit and expectation can be swapped: $\lim_{n \to \infty} E[X_n] = E[\lim_{n \to \infty} X_n]$. This is an wonderfully intuitive rule. It's like climbing a staircase; your height only ever increases, so the limit of your journey is either a specific step or infinity, and the average behaves just as predictably. For example, if we construct a random variable by summing up more and more positive terms, like $X_n = \sum_{k=1}^n U^k$ where $U$ is some random number between 0 and 1/2, the sequence $X_n$ is non-decreasing. The MCT assures us we can find the limit of the expectation by first finding the limit of $X_n$ (which becomes an infinite series) and then taking its expectation.

The real workhorse, however, is the ​​Dominated Convergence Theorem (DCT)​​. This theorem is a masterpiece of practical analysis that directly addresses the "escaping mass" problem. It gives us a condition of containment. It says that if you can find one single random variable $Y$ that acts as a universal "ceiling" for the absolute value of your entire sequence—that is, $|X_n| \le Y$ for all $n$—and this ceiling function $Y$ itself has a finite expectation (we say $Y$ is "integrable"), then you are golden. The fence is up. No mass can escape to infinity. You are free to swap the limit and expectation.

Let's see this beautiful idea in action. Imagine a measurement device whose reported value is $Y_n = n \sin(X/n)$, where $X$ is the true value of some physical quantity and $n$ is an adjustable sensitivity parameter. As we crank up the sensitivity ($n \to \infty$), we know from basic calculus that the expression gets closer and closer to $X$. So, the pointwise limit is $\lim_{n \to \infty} Y_n = X$. But can we say that the limit of the average measurement, $\lim_{n \to \infty} E[Y_n]$, is the average of the true value, $E[X]$?

To answer this, we check the DCT. We need a "fence." There is a wonderful little inequality from trigonometry: $|\sin(u)| \le |u|$ for any real number $u$. Applying this to our device gives $|Y_n| = |n \sin(X/n)| \le n \cdot |X/n| = |X|$. There it is! The random variable $|X|$ itself acts as the ceiling, the fence for our entire sequence of measurements $Y_n$. If we know that our true quantity has a finite average absolute value (i.e., $E[|X|] < \infty$), then our fence is "integrable." Both conditions of the DCT—pointwise convergence and an integrable dominator—are met. We can now confidently conclude that $\lim_{n \to \infty} E[Y_n] = E[X]$. The average of our increasingly sensitive measurements does indeed converge to the true average. The same logic applies to many functions that approximate a value, such as $n(1-\exp(-|X|/n))$ or the partial sums of a well-behaved power series like $\sum_{k=0}^n \frac{X^k}{k!}$. As long as a dominating, integrable function can be found, the interchange is valid.

What happens if we can't find such a fence? Let's return to our "moving blip" example, $X_n = n \mathbf{1}_{(0, 1/n]}$. Why does the DCT fail here? A dominating function $Y$ would have to be larger than every $X_n$. At any point $\omega$ in $(0, 1]$, $Y(\omega)$ must be larger than $n$ for all $n$ such that $\omega \le 1/n$. This implies $Y(\omega)$ must be at least as large as the function $\lfloor 1/\omega \rfloor$. But if you try to compute the integral (the expectation) of $\lfloor 1/\omega \rfloor$ from 0 to 1, you'll discover that it is infinite! No "integrable" fence exists. Our flock of values is in a pasture with no northern wall; they are free to run off towards infinity, and we can't keep track of their average position.

When domination fails, all is not lost, but we must be more cautious. For non-negative random variables, there is another famous result called ​​Fatou's Lemma​​. It's a kind of consolation prize. It states that even if mass escapes, it can't just appear out of nowhere. The limit of the average must be at least as big as the average of the limit: $\liminf_{n \to \infty} E[X_n] \ge E[\lim_{n \to \infty} X_n]$ The "limit inferior" ($\liminf$) is a technical device to handle cases where the limit of expectations might not even exist (it could oscillate). Fatou's Lemma tells us that in the process of taking the limit, value can leak out of the expectation, but it cannot be spontaneously created. Our moving blip example respects this law: $\lim_{n \to \infty} E[X_n] = 1 \ge E[\lim_{n \to \infty} X_n] = 0$.

A beautiful illustration of this principle is the "typewriter" sequence. Imagine a pulse of constant energy that, in each step $n$, scans across one of many narrow, adjacent strips that tile the interval $[0,1]$. As $n$ increases, the pulses get narrower and more intense, systematically sweeping across the whole space. For any fixed point you are watching, the pulse will eventually pass by, and the value there will drop to zero. Thus, the limit function is zero everywhere, and its expectation is zero. However, since the pulse's total energy (its expectation) is engineered to be constant at every step, the limit of the expectations is a positive number. Again, we find $\lim_{n \to \infty} E[X_n] > E[\lim_{n \to \infty} X_n]$, a classic demonstration of Fatou's Lemma and another stark reminder of the subtle dance between limits and expectations.

In the end, the question of swapping a limit and an expectation is not a mere technicality for mathematicians. It is a deep question about the stability and predictability of a system. It asks: as a process evolves, does its average behavior reflect the average of its ultimate fate? As we've seen, the answer is "only if you can keep everything contained." This principle is a cornerstone of modern probability theory, with profound implications in fields from physics and finance to statistics and engineering, where understanding the long-term average of a process is often the ultimate goal.

We have spent some time wrestling with the rather formal, mathematical machinery of limits and expectations. You might be tempted to think this is just a game for mathematicians, a matter of dotting i's and crossing t's to keep their abstract house in order. But nothing could be further from the truth! The question of when you can exchange the order of taking a limit and taking an expectation—when is $\lim \mathbb{E}[X_n] = \mathbb{E}[\lim X_n]$?—is not a mere technicality. It is a profound question whose answer unlocks the ability to build predictive models of the world across an astonishing range of disciplines. It is the bridge between the theoretical probabilities we can write down and the long-term outcomes we actually observe. Let's take a tour and see this principle at work.

The most intuitive place we see this principle is in the ​​Law of Large Numbers​​. It’s the theorem that gives mathematical teeth to our intuition about averaging. If you flip a coin many times, you feel certain that the fraction of heads will get closer and closer to $0.5$. The Strong Law of Large Numbers (SLLN) says this isn't just a feeling; it's a near-certainty. The long-term time average of independent, repeated events converges, with probability one, to their theoretical average, their expectation.

Think about the world of finance. An investor builds a portfolio with different assets, each with its own probabilities of daily gains and losses. Each day is a new roll of the dice, a chaotic and unpredictable event. But what does the investor care about? Not necessarily tomorrow's outcome, but the long-term performance. The SLLN is the tool that allows us to see through the daily noise. By calculating the expected daily return of the portfolio—a simple weighted average of the expected returns of each asset—we can, by virtue of the law, know the value to which the actual average daily return will almost surely converge over time. The limit of the real-world average is the expectation we can calculate on paper. This is the basis of any long-term investment strategy; it's the substitution of chaos for predictability, thanks to our ability to equate a limit with an expectation.

This idea becomes even more powerful when we encounter situations where the rules of the game are themselves uncertain. Imagine trying to determine the public's opinion on an issue. You can poll people one by one (a sequence of Bernoulli trials: yes or no), but what if the underlying proportion $P$ of "yes" voters in the population is unknown? In a Bayesian framework, we might model $P$ itself as a random variable drawn from some distribution, say a Beta distribution. Now, as we collect more and more data, the SLLN still works its magic. The observed frequency of "yes" votes will converge to the specific value of $P$ for that population. But since $P$ was random, the limit itself is a random variable! If we then want to find the overall expected outcome before we even start polling, we must calculate the expectation of this limit—that is, the expected value of $P^k$ for some event, averaged over the Beta distribution. Here, the law of large numbers provides the limiting object, and then we take its expectation, a beautiful two-step dance between time averages and ensemble averages.

The Law of Large Numbers is a special case of our grand theme. The more general, and more subtle, question arises when we have a sequence of different random systems. Let's say we have a sequence of random variables $X_n$ that we know converges to some limiting variable $X$. It is incredibly tempting to assume that the average of $X_n$ must therefore converge to the average of $X$. But this is a dangerous leap of faith!

This is where the great convergence theorems of analysis, like the ​​Dominated Convergence Theorem (DCT)​​ and its friendly cousin, the ​​Bounded Convergence Theorem (BCT)​​, become our license to operate. They provide the "safety conditions" under which we are allowed to swap the limit and the expectation. The core idea, put simply, is that if the random variables in your sequence can't "run away to infinity" in some pathological way—if they are collectively "dominated" by some other integrable random variable—then the swap is legal.

A classic application arises in statistics. Suppose you are sampling from a uniform distribution on $[0, \theta]$ and you don't know $\theta$. A natural guess for $\theta$ is the maximum value you've seen so far, $M_n = \max\{U_1, \dots, U_n\}$. As you take more samples ($n \to \infty$), your intuition tells you that $M_n$ will get closer and closer to the true endpoint $\theta$. Indeed, it converges almost surely to $\theta$. Now, what if we want to know the long-term expectation of some function of our estimate, say $E[f(M_n)]$? Because the function $f$ is continuous (and therefore bounded on the interval $[0, \theta]$), the BCT gives us immediate permission to swap: $\lim_{n \to \infty} \mathbb{E}[f(M_n)] = \mathbb{E}[\lim_{n \to \infty} f(M_n)] = f(\theta)$. This result is fundamental to the theory of consistent estimators.

The same principle allows us to bridge the discrete and the continuous. Consider a simple random walk, where a particle hops left or right with equal probability. The Central Limit Theorem tells us that after many steps, the particle's normalized position, $S_n/\sqrt{n}$, looks statistically like a bell curve—a standard normal distribution. But what if we want to know the expected distance from the origin, $\mathbb{E}[|S_n|/\sqrt{n}]$? Does it converge to the expected distance for a normal distribution, $\mathbb{E}[|Z|]$? To make this claim, we must justify swapping the limit and the expectation. In this case, the variables aren't uniformly bounded, so we need the full power of the DCT, which relies on a more general condition called uniform integrability. Once the swap is justified, the calculation is straightforward, and we find the limit is a universal constant, $\sqrt{2/\pi}$. This elegant result connects a simple discrete process to a fundamental constant of calculus, all hinging on the legitimacy of a limit-expectation swap.

We can see these tools work in concert. Imagine calculating the geometric mean of a sequence of random numbers, $G_n = (X_1 \cdots X_n)^{1/n}$. By taking a logarithm, this becomes an arithmetic average, and the SLLN tells us that $\ln(G_n)$ converges to some value $\mu_L$. By continuity, $G_n$ itself converges to $\exp(\mu_L)$. If we then want to find the limit of $\mathbb{E}[\arctan(G_n)]$, we can invoke the Bounded Convergence Theorem—since $\arctan(x)$ is always bounded between $-\pi/2$ and $\pi/2$—to pass the limit inside the expectation and arrive at the beautiful, simple answer: $\arctan(\exp(\mu_L))$.

The power of these ideas extends far beyond simple sequences. It allows us to build a calculus for randomness itself.

In signal processing or control theory, we often model phenomena as ​​stochastic processes​​ that evolve continuously in time, like a noisy voltage signal $X_t$. How would we even define its rate of change, $\dot{X}_t$? We must use a limit, just as in ordinary calculus. But to compute anything useful, like how the signal's value at one time correlates with its slope at another, we must compute an expectation involving this limit. The ability to swap the limit and the expectation is precisely what allows us to show that the covariance of a process with its derivative is the derivative of its covariance function: $E[X_s \dot{X}_t] = \frac{\partial}{\partial t} E[X_s X_t]$. This interchange is a foundational step in building the entire field of stochastic calculus.

Other beautiful structures emerge in probability. A ​​martingale​​ is the mathematical ideal of a "fair game"—your expected fortune tomorrow, given everything you know today, is simply your fortune today. The Martingale Convergence Theorem tells us that, under certain conditions, such processes must converge to a limiting random variable. This provides a powerful tool for analyzing systems that seem to be exploding with complexity. Consider a ​​branching process​​, which could model anything from the spread of a virus to a nuclear chain reaction. While the number of individuals in each generation may grow exponentially, it's possible to construct a related quantity—a cleverly normalized population count—that forms a martingale. Because this martingale converges and its expectation is constant, we can compute the expected value of its final limiting state simply by calculating its value at the very beginning. This is an amazing trick: a hidden stability allows for prediction amidst chaos.

Perhaps the grandest expression of this principle lies in ​​ergodic theory​​. The Birkhoff Ergodic Theorem is like the Law of Large Numbers on steroids. It applies to dynamical systems where the components are not independent, but evolve according to some deterministic or stochastic rule. It makes a breathtaking claim: for a huge class of "ergodic" systems (which are chaotic and mixing, in a specific sense), the impossibly complex long-term time average of a quantity along a single trajectory is equal to the much simpler space average (expectation) of that quantity over all possible states of the system. This is the foundational principle of statistical mechanics. It is why we can talk about the temperature of a gas (a space average over the kinetic energies of all molecules) by measuring it with a thermometer (which performs a time average at one location). Time average equals space average—the limit equals the expectation.

This is not just century-old mathematics. These ideas are workhorses on the frontiers of science. In ​​random matrix theory​​, physicists and mathematicians study the properties of large matrices whose entries are random variables. The eigenvalues of these matrices describe an incredible variety of phenomena, from the energy levels in heavy atomic nuclei to the structure of complex networks like the internet. A key tool is the Stieltjes transform, which packages the information about all the eigenvalues into a single function $s_N(z)$. A cornerstone result, Wigner's semicircle law, states that as the matrix size $N$ goes to infinity, $s_N(z)$ converges to a specific, non-random function $s_c(z)$. To find the average behavior of the system, we need to know $\lim_{N \to \infty} \mathbb{E}[s_N(z)]$. Is it simply $s_c(z)$? Yes, and the justification comes directly from the Bounded Convergence Theorem, as the Stieltjes transform is nicely bounded away from the real axis.

So, we see that our initial, seemingly technical question is at the very heart of how we connect theory to observation. It is what guarantees that the laws of probability translate into predictable long-term behavior, whether for an investor's portfolio, the evolution of a physical system, or the fundamental properties of matter itself. It is a golden thread that runs through the fabric of modern science.