Transformation of Random Variables

In fields from physics to finance, the raw data we collect is often just the starting point. We are frequently more interested in a new quantity that is a function of our original measurement—such as calculating kinetic energy from velocity, or determining the magnitude of an error from the error measurement itself. This process gives rise to a "transformed random variable." But this raises a fundamental question: if we know the probability distribution of the original variable, how can we determine the distribution of the new, transformed one? Answering this is not just a mathematical exercise; it is the key to unlocking deeper insights and building more powerful models.

This article provides a comprehensive guide to understanding and applying the transformation of random variables. The first chapter, ​​"Principles and Mechanisms,"​​ will lay the groundwork, starting with the core logic of transforming discrete and continuous variables. We will explore powerful techniques like the change-of-variables formula, the universally applicable CDF method, and the elegant Moment Generating Function approach. The second chapter, ​​"Applications and Interdisciplinary Connections,"​​ will bridge theory and practice. We will see how these transformations are used to scale data, forge new distributions, and solve real-world problems in data science, physics, information theory, and more, revealing the hidden connections that unify the scientific world.

Imagine you are a scientist studying a phenomenon. You've collected a mountain of data, which you've modeled with a random variable, let's call it $X$. This variable has a certain personality, a probability distribution that tells you which outcomes are likely and which are rare. But often, the raw data isn't the end of the story. You might be interested in a different quantity that depends on your original measurement. For instance, if $X$ is the velocity of a particle, you might be more interested in its kinetic energy, which is proportional to $X^2$. Or, if $X$ is the error in a measurement, you might only care about the magnitude of the error, $|X|$.

In each case, you are creating a new random variable, let's call it $Y$, by applying a mathematical function, $g$, to your original variable: $Y = g(X)$. The immediate, and fascinating, question is: if we know the life story of $X$—its probability distribution—can we deduce the life story of $Y$? The answer is a resounding yes, and the process of doing so is a beautiful journey through the logic of probability. We are not just manipulating symbols; we are translating one probabilistic story into another.

Let's begin in the simplest setting: the world of discrete random variables, where outcomes are countable, like the number of dots on a pair of dice. Suppose our original variable $X$ can only take on a specific set of values. Now, we apply a function to it, say $Y = g(X)$. How do we find the probability of a particular outcome for $Y$, say $Y=y$?

The logic is beautifully simple. We just have to play a game of "find and gather." We look back at all the possible outcomes of our original variable $X$. Which of them, when plugged into our function $g(X)$, produce the value $y$? Let's say we find a few of them: $x_1, x_2, \dots, x_k$. Since these are distinct outcomes for $X$, they are mutually exclusive events. Therefore, the total probability of getting $Y=y$ is simply the sum of the probabilities of all these "pre-image" outcomes. In mathematical terms, the probability mass function (PMF) of $Y$ is: $p_Y(y) = P(Y=y) = \sum_{x \text{ such that } g(x)=y} P(X=x)$

Consider a simple sensor whose output $X$ is one of the integers $\{-2, -1, 0, 1, 2\}$, each with equal probability of $\frac{1}{5}$. A post-processing unit computes a new signal $Y = X^2+1$ to amplify its magnitude. What is the PMF of $Y$?

Let's follow the procedure. The possible values for $Y$ are:

• If $X=0$, then $Y = 0^2+1=1$.
• If $X=1$ or $X=-1$, then $Y = (\pm 1)^2+1=2$.
• If $X=2$ or $X=-2$, then $Y = (\pm 2)^2+1=5$.

Now we gather the probabilities.

• The only way to get $Y=1$ is if $X=0$. So, $p_Y(1) = p_X(0) = \frac{1}{5}$.
• To get $Y=2$, $X$ could have been either $-1$ or $1$. So, we add their probabilities: $p_Y(2) = p_X(-1) + p_X(1) = \frac{1}{5} + \frac{1}{5} = \frac{2}{5}$.
• Similarly, to get $Y=5$, $X$ could have been $-2$ or $2$. So, $p_Y(5) = p_X(-2) + p_X(2) = \frac{1}{5} + \frac{1}{5} = \frac{2}{5}$.

Notice what happened. The transformation was not one-to-one; multiple values of $X$ were mapped to the same value of $Y$. This caused the probabilities to "bunch up," making $Y=2$ and $Y=5$ twice as likely as $Y=1$. This simple principle of identifying pre-images and summing their probabilities is the fundamental mechanism for all discrete transformations.

What happens when we move to the continuous world? Here, $X$ can take any value in a range, and the probability of any single point is zero. We can no longer sum probabilities. Instead, we must think about probability density, which you can visualize as the "heaviness" or "concentration" of probability at different points.

The guiding principle is the ​​conservation of probability​​. Imagine a tiny interval of width $dx$ around a point $x$. The probability that our variable $X$ falls into this interval is approximately $f_X(x)dx$, where $f_X(x)$ is the probability density function (PDF) of $X$. Our transformation $y = g(x)$ maps this tiny interval $dx$ to a new tiny interval $dy$. The probability must be conserved: the probability mass that was in $dx$ must now be in $dy$. $f_Y(y)|dy| = f_X(x)|dx|$ We use absolute values because area and density must always be positive. A simple rearrangement gives us the famous ​​change-of-variables formula​​: $f_Y(y) = f_X(x) \left| \frac{dx}{dy} \right|$ Where $x$ must be expressed in terms of $y$ (i.e., $x = g^{-1}(y)$). This formula tells us that the new density $f_Y(y)$ is the old density $f_X(x)$ scaled by a factor $|\frac{dx}{dy}|$. This factor represents how much the transformation stretches or compresses the space. If an interval is stretched, its density must decrease to keep the probability the same. If it's compressed, its density must increase.

Let's see this in action. A chi-squared distribution with one degree of freedom, $X \sim \chi^2(1)$, models the energy of a random signal. Its PDF is $f_X(x) = \frac{1}{\sqrt{2\pi x}} \exp(-\frac{x}{2})$ for $x \gt 0$. Suppose we want to find the distribution of the signal's amplitude, which would be $Y = \sqrt{X}$.

Here, our transformation is $g(x) = \sqrt{x}$. The inverse is $x = y^2$. The scaling factor is the derivative of the inverse function: $\frac{dx}{dy} = 2y$. Since $Y$ represents an amplitude, we are interested in $y \gt 0$, so $|\frac{dx}{dy}| = 2y$. Plugging everything into our formula: $f_Y(y) = f_X(y^2) \cdot |2y| = \frac{1}{\sqrt{2\pi y^2}} \exp\left(-\frac{(y^2)}{2}\right) \cdot (2y)$ $f_Y(y) = \frac{1}{\sqrt{2\pi} \cdot y} \exp\left(-\frac{y^2}{2}\right) \cdot (2y) = \sqrt{\frac{2}{\pi}} \exp\left(-\frac{y^2}{2}\right), \text{ for } y \gt 0$ This resulting distribution is known as a half-normal distribution. The transformation has taken the energy distribution and given us the corresponding amplitude distribution, all through a simple rule of density scaling.

The change-of-variables formula is slick, but it relies on the function $g(x)$ being one-to-one (monotone), so that the inverse $g^{-1}(y)$ is well-defined. What if it's not? What about a function like $y=x^2$ where $x$ can be positive or negative?

We need a more robust, more fundamental approach. And there is one: the ​​Cumulative Distribution Function (CDF) method​​. It is foolproof and works for any transformation. The logic is to always start from the basic definition of the CDF: $F_Y(y) = P(Y \le y)$ Then, substitute $Y=g(X)$ and manipulate the inequality to isolate $X$. $F_Y(y) = P(g(X) \le y)$ Once we have an expression in terms of $X$, we can use the known CDF or PDF of $X$ to calculate the probability. If we need the PDF of $Y$, we can simply differentiate the CDF we found: $f_Y(y) = \frac{d}{dy} F_Y(y)$.

Let's take the problem of measuring the magnitude of a positional error, $Y=|X|$, where the error $X$ is uniformly distributed on $[-A, A]$. The transformation $g(x)=|x|$ is not one-to-one. Let's find the CDF of $Y$ for a value $y$ between $0$ and $A$: $F_Y(y) = P(Y \le y) = P(|X| \le y) = P(-y \le X \le y)$ Since $X$ is uniform on $[-A, A]$, its PDF is $f_X(x) = \frac{1}{2A}$. The probability of falling in the interval $[-y, y]$ is its length, $2y$, times the density: $F_Y(y) = \int_{-y}^{y} \frac{1}{2A} dx = \frac{2y}{2A} = \frac{y}{A}$ So, the CDF for $y \in [0, A]$ is simply $F_Y(y)=\frac{y}{A}$. Differentiating this gives the PDF: $f_Y(y) = \frac{1}{A}$ for $y \in [0, A]$. The probability density from the negative axis has been "folded over" and added to the positive axis, doubling the density (from $\frac{1}{2A}$ to $\frac{1}{A}$) on half the interval.

This method shines with even more complex functions. Imagine modeling the phase $X$ of a random signal as a uniform variable on $[0, 2\pi]$. What's the distribution of its measured amplitude, $Y=\cos(X)$? Intuitively, a point moving at a constant angular speed on a circle has a horizontal projection (the cosine) that moves fastest through the center and lingers near the endpoints. So we'd expect the probability density of $Y$ to be highest near $-1$ and $1$. Let's check with the CDF method for $y \in [-1, 1]$. $F_Y(y) = P(\cos(X) \le y)$ On the interval $[0, 2\pi]$, the inequality $\cos(x) \le y$ is true for $x$ in the range $[\arccos(y), 2\pi - \arccos(y)]$. Since $X$ is uniform on $[0, 2\pi]$, the probability is the length of this interval divided by $2\pi$: $F_Y(y) = \frac{(2\pi - \arccos(y)) - \arccos(y)}{2\pi} = 1 - \frac{\arccos(y)}{\pi}$ Differentiating gives the PDF: $f_Y(y) = \frac{1}{\pi\sqrt{1-y^2}}$ for $y \in (-1,1)$. This function blows up at $y=-1$ and $y=1$, just as our intuition predicted! The linger-time is indeed longest at the turning points.

Among all possible transformations, there is one that is so special and profound it feels like a magic trick. It's called the ​​Probability Integral Transform​​. It states that for any continuous random variable $X$ with CDF $F_X(x)$, the new random variable defined by the transformation $Y = F_X(X)$ will have a uniform distribution on the interval $[0, 1]$. Let's prove this with the CDF method we just learned. Let's find the CDF of $Y=F_X(X)$. For any $y$ between 0 and 1: $F_Y(y) = P(Y \le y) = P(F_X(X) \le y)$ Since the CDF $F_X$ is a non-decreasing function, we can apply its inverse $F_X^{-1}$ to both sides of the inequality: $F_Y(y) = P(X \le F_X^{-1}(y))$ But the definition of the CDF is exactly this! $P(X \le z) = F_X(z)$. So, we have: $F_Y(y) = F_X(F_X^{-1}(y)) = y$ The CDF of $Y$ is $F_Y(y) = y$ for $y \in [0, 1]$. This is the CDF of a uniform distribution on $[0, 1]$! This result is stunningly general. It doesn't matter how weird or complicated the original distribution of $X$ is; when seen through the "lens" of its own CDF, it looks perfectly flat and uniform. This principle is the theoretical foundation for simulation and a cornerstone of modern statistics, as it gives us a way to turn standard uniform random numbers (which computers can generate easily) into random numbers from any distribution we desire. It can also appear in disguise, such as in the transformation $Y=\exp(-X/2)$ for a variable $X \sim \chi^2(2)$, which also surprisingly results in a uniform distribution.

So far, we have attacked the problem head-on, working directly with PMFs and PDFs. But sometimes in science, the most elegant path is an indirect one. Enter the ​​Moment Generating Function (MGF)​​. The MGF of a random variable $X$, written $M_X(t)$, is defined as $M_X(t) = \mathbb{E}[\exp(tX)]$. It's a "transform" of the distribution, much like a Fourier or Laplace transform. Its power comes from two facts:

1. The MGF, if it exists, uniquely determines the distribution. If two variables have the same MGF, they have the same distribution.
2. MGFs have some wonderful properties that make certain problems incredibly simple.

The most famous of these properties relates to linear transformations. If we have a new variable $Y = a X + b$, finding its PDF directly can be tedious. But finding its MGF is trivial: $M_Y(t) = \mathbb{E}[\exp(t(aX+b))] = \mathbb{E}[\exp(atX) \exp(bt)] = \exp(bt) \mathbb{E}[\exp((at)X)]$ Which gives the beautiful rule: $M_Y(t) = \exp(bt) M_X(at)$ For example, if the lifetime $X$ of an LED has an MGF of $M_X(t) = (1 - \frac{t}{4})^{-2}$ and we define a new variable $Y = 5X - 3$, we don't need to know anything else about the distribution of $X$ to find the MGF of $Y$. We just apply the rule with $a=5$ and $b=-3$: $M_Y(t) = \exp(-3t) M_X(5t) = \exp(-3t) \left(1 - \frac{5t}{4}\right)^{-2}$ We have found the MGF of $Y$ in one line. If we recognized this new MGF as belonging to a known distribution, we would have found the distribution of $Y$ without ever touching a PDF or a CDF. This method allows us to operate in a different mathematical space where transformations become simple multiplications and shifts.

Our journey has focused on transforming a single random variable. But what if our new variable is a function of several random variables? For instance, $Z = \max(X, Y)$ or $Z = X+Y$. The same core principles apply, but now we must navigate a multi-dimensional space.

In the discrete case, if we want to find $P(Z=z)$, we must search the entire grid of possible $(x, y)$ pairs and sum the joint probabilities $p(x, y)$ for all pairs that satisfy the condition $g(x, y) = z$. For a continuous case with $Z=g(X,Y)$, finding the CDF $F_Z(z) = P(Z \le z)$ involves integrating the joint PDF $f(x, y)$ over the entire region in the $xy$-plane where the inequality $g(x,y) \le z$ holds true.

This step into multiple dimensions opens up a vast and rich field of study, leading to central concepts like the distribution of sums of independent variables and the famous Central Limit Theorem. The logical tools remain the same: identify the event in the source space and calculate its total probability. The art and the beauty lie in seeing how these fundamental principles scale up, allowing us to understand the intricate web of probabilities that governs our complex world.

After our journey through the essential machinery of transforming random variables, you might be wondering, "What is all this for?" It's a fair question. It's one thing to be able to turn the crank on the mathematical formulas, but it's another thing entirely to see why anyone would want to. The beauty of this subject, like so much of physics and mathematics, is not just in the "how" but in the "why." It's about learning to see the world through different lenses.

Sometimes, you need a magnifying glass; other times, a telescope. Sometimes, you need glasses that turn everything upside down. A transformation of a random variable is exactly this: a new lens. We aren't changing the underlying phenomenon, but we are changing our description of it to reveal something new, to make a hidden pattern visible, or to connect it to a different part of the scientific landscape. In this chapter, we will explore this art of "reshaping reality," seeing how these transformations bridge disciplines from finance and physics to data science and information theory.

The most straightforward transformations are the linear ones: stretching, shifting, and scaling a variable, much like converting temperature from Celsius to Fahrenheit. If you know the uncertainty (the variance) of the daily temperature in Celsius, you can immediately find the variance in Fahrenheit without re-running years of measurements. The relationship $Var(a + bX) = b^2 Var(X)$ is the precise mathematical statement of this intuition. The shift $a$ doesn't change the spread at all (shifting all your data points by 5 doesn't make them more spread out), but the scaling factor $b$ stretches or shrinks the number line, and since variance is measured in squared units, its effect goes as $b^2$.

Most probability distributions, when you stretch or shift them, become a scaled version of their old selves. But some are special. The peculiar Cauchy distribution, a wild beast in the menagerie of probabilities, has the remarkable property of being "stable." If you take a Cauchy-distributed variable, and then stretch and shift it, what you get is another Cauchy distribution. It's as if a photograph of a cat, when zoomed in and cropped, revealed another, different-looking cat. This stability is rare and points to a kind of self-contained world that the Cauchy distribution lives in.

This idea of scaling reveals something truly profound when we look at stochastic processes, which unfold in time. Consider the random, jittery path of a pollen grain in water—Brownian motion, mathematically described by the Wiener process, $W_t$. At any time $t$, the position of the particle has a normal distribution with a variance that grows linearly with time, $t$. Now, what happens if we "normalize" our view by scaling the position by $\frac{1}{\sqrt{t}}$? We define a new variable $Z = W_t / \sqrt{t}$. We find that $Z$ always has the exact same standard normal distribution, no matter what time $t$ we choose. This is a manifestation of a deep physical principle: self-similarity. A random walk looks statistically the same whether you watch it for one second or for one hour, as long as you scale your viewing window appropriately. This single transformation uncovers a fractal-like symmetry hidden within the heart of randomness.

Now we move beyond simple scaling into the realm of true alchemy, where we can forge entirely new kinds of distributions from old ones. These non-linear transformations can drastically change the shape and meaning of a variable.

Imagine you are modeling the market share of a product, a proportion $P$ that must live between 0 and 1. The Beta distribution is a wonderfully flexible tool for this. But what if you are interested in a related question: how does the wealth of companies, which can grow to enormous sizes, distribute itself? It turns out that a simple transformation can connect these two worlds. If $X$ follows a specific Beta distribution (modeling a proportion close to 1), the new variable $Y = (1-X)^{-1}$ follows a Pareto distribution. The Pareto distribution is famous for describing phenomena where a small number of events account for a large part of the outcome—the "80-20 rule." This transformation shows us a hidden mathematical bridge between the world of bounded proportions and the "heavy-tailed" world of extreme events. It is a striking example of the unity of probability theory.

In modern data science, perhaps no transformation is more vital than the ​​logit​​. Many models, like linear regression, are built to predict outcomes on the entire number line, from $-\infty$ to $+\infty$. But what if you want to predict a probability, like the chance of a patient responding to a treatment? Such a probability $P$ is stubbornly stuck in the interval $(0,1)$. How do you connect the boundless world of a linear model to the confined world of probability? The logit transformation is the magical bridge: $L = \ln\left(\frac{P}{1-P}\right)$. This function takes any number from $(0,1)$ and stretches it onto the entire real line. The quantity $\frac{P}{1-P}$ is the "odds," so the logit is the "log-odds." By having a model predict $L$ instead of $P$, we can use the powerful tools of linear modeling and then transform the result back to a probability. This very idea is the foundation of logistic regression, a workhorse of fields ranging from epidemiology to finance.

The logarithm is a recurring hero in the story of data transformation. Why? Because many processes in nature are multiplicative. Population growth, investment returns, radioactive decay—these things compound. By taking the logarithm, we turn these multiplicative processes into additive ones, which are often far easier to analyze. The logarithm acts like a Rosetta Stone, translating a difficult language into a simpler one.

Consider the Gamma distribution, which often models waiting times or the accumulation of random events. Data from a Gamma distribution can be highly skewed, with a long tail to the right. This skewness can cause problems for many statistical methods. Taking the natural log of a Gamma-distributed variable, $Y = \ln(X)$, gives you a new distribution known as the log-gamma. This transformation can "tame" the skewness, making the data more symmetric and the underlying patterns more visible. It's like putting on the right pair of prescription glasses.

Similarly, the F-distribution, the cornerstone of the Analysis of Variance (ANOVA) used to compare different groups, is also skewed. It represents a ratio of variances. By transforming it with a logarithm, $Y = \ln(X)$, we again create a more symmetric distribution that is often more amenable to modeling. In countless fields, scientists and engineers take logs of their data not as a mindless ritual, but as a purposeful transformation to better reveal the underlying structure.

A transformation doesn't have to be a smooth mathematical formula. It can be any well-defined rule that maps inputs to outputs. For example, a public health agency might take detailed air quality data ('Good', 'Moderate', 'Unhealthy') and transform it into a simpler public alert system ('Good', 'Advisory'). This is a function: $Y = g(X)$, where $g(\text{'Moderate'}) = \text{'Advisory'}$ and $g(\text{'Unhealthy'}) = \text{'Advisory'}$.

What is the effect of such a transformation? We simplify the message, but we lose information. We can quantify this using the concept of Shannon entropy. By grouping outcomes, the number of possibilities decreases, and the total uncertainty, or entropy, of the system is reduced. This illustrates a fundamental tradeoff in all of science and communication: the balance between simplicity and detail. Every time we create a model or summarize data, we are performing a transformation that, by its very nature, discards some information to highlight another.

Finally, we arrive at the most powerful and abstract transformation of all: the Fourier transform. In probability, this is known as the characteristic function. It transforms a probability density function from its natural "value space" into a "frequency space." Why would we do this? Because sometimes a horribly complicated problem in one space becomes astonishingly simple in the other.

Consider the task of finding the distribution of $Y=X^2$. A direct approach can be cumbersome. But if we move to the Fourier world, we can find an elegant solution. The final PDF for $Y$ can be expressed as an integral involving the characteristic function of $X$ and a cosine term, $\cos(k\sqrt{y})$. The appearance of the cosine is no accident. The transformation $Y=X^2$ is symmetric ($x$ and $-x$ both map to the same $y$), and the cosine is a symmetric (even) function. The symmetry in the original transformation is reflected as a symmetry in its Fourier representation. This is a deep and beautiful principle. This technique allows physicists and engineers to solve problems in wave mechanics, signal processing, and quantum mechanics by jumping into this abstract space, performing a simple multiplication or shift, and then jumping back to the "real world" with the solution in hand.

From changing units to uncovering the fractal nature of randomness, from forging new statistical tools to quantifying information itself, the transformation of random variables is not just a chapter in a textbook. It is a fundamental way of thinking, a versatile and powerful toolkit for seeing the hidden connections that unify the scientific world.