Information Diagrams

Understanding complex systems, from communication networks to artificial intelligence, requires grasping the intricate relationships between their components. Information theory offers a powerful mathematical framework for this, quantifying concepts like uncertainty and shared knowledge. However, its abstract formulas can obscure the elegant, underlying truths. This article introduces Information Diagrams, a visual language that bridges this gap by translating the mathematics of information into intuitive, geometric pictures. By exploring this visual grammar, you will gain a deeper understanding of the core principles of information theory and its profound impact across various disciplines. The journey begins with the foundational "Principles and Mechanisms" that govern these diagrams, followed by a tour of their "Applications and Interdisciplinary Connections," revealing how simple circles can illuminate complex problems in science and engineering.

Imagine you are trying to understand a complex machine with several moving parts. You could study each part in isolation, but that wouldn't tell you how they work together. The real magic happens at the interfaces, in the ways the parts influence, constrain, and inform one another. Information theory gives us a language to talk about these relationships, not just for machines, but for any system where uncertainty and knowledge are at play. And like any good language, it has a visual grammar: the ​​information diagram​​.

These diagrams, which look much like the Venn diagrams you remember from school, are more than just pretty pictures. They are a powerful tool for building intuition, for turning abstract formulas into tangible geometric truths. Let's embark on a journey to explore this visual language, starting with the simplest elements and building our way up to a rich, descriptive grammar of information.

Let's start with two random variables, let's call them $X$ and $Y$. Think of $X$ as the outcome of a coin flip and $Y$ as the weather tomorrow. Each variable carries some amount of uncertainty, or "surprise." In information theory, we give this a name: ​​entropy​​, denoted $H(X)$. You can think of the entropy $H(X)$ as the total "area" of all possible information contained within the variable $X$. In our diagram, we will represent $H(X)$ as a circle. The larger the circle, the more uncertain the variable.

Now, what happens when we have two variables, $X$ and $Y$? We draw two circles. If these variables have something to do with each other, the circles will overlap. This overlapping region is the heart of the matter. It represents the information that is common to both $X$ and $Y$. This is the ​​mutual information​​, denoted $I(X;Y)$. It's what you learn about $X$ by observing $Y$, and vice versa.

How can we define this overlap? One way is to think about how much our uncertainty about $X$ is reduced once we know $Y$. We start with the total uncertainty in $X$, which is $H(X)$, and we subtract the uncertainty that remains in $X$ even after we know $Y$. This remaining uncertainty is called the ​​conditional entropy​​, $H(X|Y)$. What's left must be the information that $Y$ provided about $X$. So, we can write:

$I(X;Y) = H(X) - H(X|Y)$

This corresponds to taking the whole circle for $X$ and removing the part that doesn't overlap with $Y$. What remains is, of course, the intersection.

But here is where the diagram reveals a beautiful symmetry. We could have started with $Y$. The information $X$ provides about $Y$ is the total uncertainty in $Y$ minus the uncertainty that remains after we know $X$:

$I(Y;X) = H(Y) - H(Y|X)$

Looking at our diagram, we see that both calculations—$H(X) - H(X|Y)$ and $H(Y) - H(Y|X)$—point to the exact same, single region: the intersection. The diagram makes it visually obvious that $I(X;Y) = I(Y;X)$. The information that $X$ has about $Y$ is identical to the information that $Y$ has about $X$. This symmetry, which can seem a bit abstract in equations, becomes a simple, undeniable geometric fact.

The parts of the circles that don't overlap also have meaning. The part of the $X$ circle outside of $Y$ is precisely that remaining uncertainty, $H(X|Y)$. It's the information that is unique to $X$. Symmetrically, the part of the $Y$ circle outside of $X$ is $H(Y|X)$.

The power of a good model is often revealed at its extremes. What do our diagrams look like for systems with very simple relationships?

First, consider two variables that are completely unrelated, or ​​statistically independent​​. Imagine flipping a coin in New York ($X$) and another in Tokyo ($Y$). The outcome of one tells you absolutely nothing about the other. There is no shared information. How would we draw this? The circles for $H(X)$ and $H(Y)$ would not overlap at all. Their mutual information, $I(X;Y)$, is zero. In this case, the total uncertainty of the combined system, the ​​joint entropy​​ $H(X,Y)$, is simply the sum of the individual uncertainties: $H(X,Y) = H(X) + H(Y)$. The diagram shows this additivity perfectly: the total area is just the sum of the two separate areas.

Now, let's consider the opposite extreme: a ​​deterministic relationship​​. Suppose you roll a six-sided die ($X$) and we define a second variable $Y$ to be simply "is the outcome even or odd?". Once you know the result of the die roll (say, $X=4$), you know the value of $Y$ with absolute certainty ($Y=\text{even}$). There is zero uncertainty left in $Y$ once $X$ is known. This means the conditional entropy $H(Y|X)$ is zero.

How does our diagram represent this? If $H(Y|X)$ is the part of the $Y$ circle outside the $X$ circle, and that area must be zero, then the only possibility is that the entire circle for $Y$ is contained within the circle for $X$. This is a beautiful visual! It shows that all the information in $Y$ was already present in $X$. Knowing the specific number on the die is a finer-grained piece of information than just knowing its parity. And because the $Y$ circle is completely inside the $X$ circle, their intersection (the mutual information) is simply the entire $Y$ circle. That is, $I(X;Y) = H(Y)$. All the information in $Y$ is mutual information with $X$.

The world is rarely as simple as two variables. Let's introduce a third, $Z$, and its corresponding circle. The diagram now has three overlapping circles, creating a tapestry of seven distinct regions. The total area covered by the union of all three circles represents the total uncertainty of the entire system, the joint entropy $H(X,Y,Z)$.

This richer diagram allows us to visualize more subtle and powerful ideas. One of the most fundamental principles in information theory is that "knowing more cannot increase uncertainty." Formally, this is written as the inequality $H(X|Y) \ge H(X|Y,Z)$. It means that the uncertainty you have about $X$ when you know $Y$ must be greater than or equal to the uncertainty you have about $X$ when you know both $Y$ and $Z$. The formula is a bit of a mouthful, but the diagram makes it trivial.

$H(X|Y)$ is the area of the $X$ circle that lies outside the $Y$ circle. Now, $H(X|Y,Z)$ is the area of the $X$ circle that lies outside both the $Y$ and $Z$ circles. It is immediately obvious from the picture that the second region is a part of the first one. You can't add area by removing more of the circle! Thus, the inequality must hold. The diagram has turned a formal proof into a simple act of seeing.

This brings us to one of the most useful concepts the three-variable diagram can illustrate: ​​conditional mutual information​​. This quantity, written $I(X;Y|Z)$, asks, "How much information do $X$ and $Y$ share, given that we already know Z?" Imagine $X$ is a child's shoe size, $Y$ is their reading ability, and $Z$ is their age. In the general population, shoe size and reading ability are correlated—older children have bigger feet and read better. But if we look only at a group of 8-year-olds (conditioning on $Z=8$), that correlation largely vanishes.

The information diagram gives us a stunningly clear picture of this. $I(X;Y|Z)$ is the part of the overlap between $X$ and $Y$ that is outside the $Z$ circle. It's the information shared between $X$ and $Y$ that is not explained away by $Z$. When we say $X$ and $Y$ are ​​conditionally independent​​ given $Z$, we are making the formal statement $I(X;Y|Z)=0$. In our diagram, this simply means that the region for $I(X;Y|Z)$ has zero area. Any overlap between $X$ and $Y$ must occur inside the $Z$ circle.

This is the beauty of information diagrams. They take the abstract, and sometimes intimidating, mathematics of information and map it onto a visual space where our powerful geometric intuition can take over. They reveal the hidden symmetries and nested relationships of information, not as a series of theorems to be proven, but as a landscape to be explored.

Now that we have acquainted ourselves with the basic grammar of information diagrams—how to draw them and what the different areas signify—we can begin to see their true power. These are not merely neat bookkeeping devices for entropy; they are a veritable lens through which we can view the world. By translating problems from engineering, computer science, and even the philosophy of science into this visual language, we often find that complex, seemingly unrelated questions share a common, beautiful structure. We are about to embark on a short tour of these applications, to see how the simple act of drawing circles and measuring their overlap can grant us profound insights into the workings of communication, learning, and discovery itself.

Information theory was born out of the very practical problems of communication: how to send messages efficiently and reliably. It is only natural that our tour begins here, in the discipline that these diagrams call home.

First, consider the art of forgetting. Every time you save a JPEG image, stream a video, or listen to an MP3 file, you are benefiting from a process called lossy compression. The goal is to make the data smaller, but this comes at a cost—a loss of perfect fidelity. There is a fundamental trade-off between the rate (how many bits we use to describe the data) and the distortion (how much quality we lose). How can we visualize this bargain?

Let's imagine our original data is a variable $X$, and its compressed reconstruction is $\hat{X}$. The information diagram for these two variables tells the whole story. The rate of our code, the amount of information about $X$ that is successfully transmitted, corresponds to the mutual information $I(X;\hat{X})$—the area where the two circles overlap. The remaining uncertainty we have about the original source, even after seeing its reconstruction, is the conditional entropy $H(X|\hat{X})$. This is the part of the $X$ circle that does not overlap with $\hat{X}$, and it represents the unavoidable ambiguity or distortion introduced by the compression.

Now, a fascinating question arises: what happens at the breaking point? Imagine we are compressing a signal and we reduce the transmission rate lower and lower, until it is just barely above zero. At this critical threshold, we are on the verge of getting no information at all. What does the information diagram look like? One might guess that the loss of information is a symmetric, graceful process. But the diagrams reveal a surprising and beautiful asymmetry. In this limit, virtually all the "unshared" information is concentrated in the $H(X|\hat{X})$ region. The other piece of unshared information, the "reconstruction noise" $H(\hat{X}|X)$, shrinks to zero. This means that at the edge of failure, the problem isn't that the code is "adding noise"; the problem is that we are left almost completely guessing what the original source was. The diagram shows us that the communication channel becomes perfectly one-sided in its failure mode: all ambiguity, no noise.

Now, let's turn from the art of forgetting to its opposite: the art of remembering. When we send information across a noisy channel—from a deep-space probe back to Earth, for instance—our goal is to protect it from corruption. This is the world of error-correcting codes. Modern codes, like the ones that power your smartphone's 5G connection, use a wonderfully clever iterative process. You can think of the decoder as a team of detectives working on a case. One detective looks at one clue, forms a hypothesis, and passes a "soft" message—not a firm conviction, but a level of belief—to the next detective. This second detective combines that message with their own clue and passes an updated belief to another, and so on. They pass these messages back and forth, hoping to converge on the truth.

How can we be sure this committee of detectives will ever reach a consensus? Information theory provides the answer. We can measure the "amount of information" contained in each soft message, quantified by the mutual information between the message and the unknown truth. The analysis of these systems, using tools like EXIT charts, is a form of information accounting. It tracks how information flows and accumulates within the decoder. For instance, the information a decoder has about a particular bit is the sum of the information it got from the a priori beliefs of its colleagues, the direct evidence from the noisy channel, and the "extrinsic" information it generated itself by using the code's structure. The principle here is that information from independent sources combines in a very powerful way, allowing the iterative process to bootstrap itself from near-total uncertainty to near-certainty. The flow of information becomes a tangible, trackable quantity that determines whether the code will succeed or fail.

The ideas of information flow have found a powerful new application in the field of machine learning and artificial intelligence. One of the central goals of modern AI is to learn "representations"—to distill raw, high-dimensional data like an image into a compact, useful summary. What makes a summary useful? It should tell us what we want to know, and nothing more.

The Information Bottleneck (IB) principle formalizes this intuition using our familiar diagrams. Imagine we have an input $X$ (say, a picture of an animal) and we want to predict a target variable $Y$ (the species, e.g., "cat" or "dog"). A machine learning model learns a compressed representation, or summary, $T$ of the input. The information diagram for the three variables $X$, $Y$, and $T$ becomes our map.

The goal is twofold. First, we want our summary $T$ to be as informative as possible about the target $Y$. This means we want to maximize the overlap between their circles, the mutual information $I(T;Y)$. Second, we want the summary to be simple, to discard all the irrelevant details of the input image (like the background color, the lighting, the specific pose of the cat). This means we want to make the summary $T$ as small as possible by minimizing its overlap with the input $X$, the mutual information $I(X;T)$.

The information diagram lays this trade-off bare. The ideal representation $T$ would be one that perfectly captures the "relevant information" $I(X;Y)$ while discarding everything else. In the three-variable diagram, there is a specific region corresponding to $I(X;T|Y)$. This is the information that our summary $T$ has learned from the input $X$ that is completely irrelevant for predicting the target $Y$. The goal of an ideal learning algorithm, according to the IB principle, is to squeeze this region of irrelevant information down to zero. Learning, in this view, is an act of information-theoretic compression: forcing the rich information from the world through the bottleneck of relevance.

Finally, we arrive at the most foundational use of these diagrams: as a tool for scientific reasoning itself. How do we make sense of data? And how can we distinguish mere correlation from true causation?

Consider a simple scenario. A scientist is studying a phenomenon $X$. She takes a measurement, call it $Y_1$. Then, she processes this measurement—perhaps by smoothing it, or running it through an algorithm—to produce a second dataset, $Y_2$. We have a chain of events: $X \rightarrow Y_1 \rightarrow Y_2$. Does the processed data $Y_2$ tell her anything new about the original phenomenon $X$ that wasn't already in $Y_1$? Common sense might suggest "maybe," but the information diagram gives a definitive "no."

Because $Y_2$ is created solely from $Y_1$ without any further access to $X$, a fundamental rule called the Data Processing Inequality comes into play. Visually, the circle for $Y_2$ can only contain a subset of the information present in $Y_1$. Therefore, the information that $Y_2$ shares with $X$ must be less than or equal to the information $Y_1$ shares with $X$. That is, $I(X;Y_2) \leq I(X;Y_1)$. More strongly, any information the pair $(Y_1, Y_2)$ provides about $X$ is exactly the same as the information $Y_1$ provides on its own: $I(X; Y_1, Y_2) = I(X; Y_1)$. Processing data cannot create new information. This might seem obvious, but it is a profoundly important principle in statistics and science, and the information diagram makes its truth self-evident.

This leads us to the final, and perhaps deepest, application: distinguishing correlation from causation. We are often told that "correlation does not imply causation," but why? Let's consider a classic case: we observe that sales of ice cream ($X$) are correlated with incidents of drowning ($Y$). Does eating ice cream cause drowning? Of course not. There is a common cause: hot weather ($Z$). When it's hot, more people buy ice cream, and more people go swimming (and thus, tragically, more drownings occur). This is a common cause structure: $X \leftarrow Z \rightarrow Y$.

In the observational world, the information diagram for $X$ and $Y$ shows an overlap, $I(X;Y) > 0$. They share information because they both inherit it from the common cause, $Z$. But what if we could perform an intervention? Imagine we could magically make the weather cold, but still force everyone to buy ice cream. In this new, intervened world, the causal link from weather to ice cream sales is broken. The common cause is gone. What happens to the information diagram? The variables $X$ and $Y$ become independent; their circles pull apart, and their mutual information becomes zero.

The amount by which the mutual information decreased, from what we observed to what happened after our intervention, is a precise measure of the "spurious" correlation induced by the common cause. Information diagrams thus provide a rigorous language to reason about not just the world as we see it, but about the causal webs that structure it. They allow us to quantify the difference between watching the world and changing it.

From the engineering of a data packet, to the learning process of an AI, to the very logic of scientific discovery, the consistent and beautiful language of information diagrams reveals the hidden unity between these domains. They are a testament to the idea that information is a universal currency, and its flow, transformation, and conservation govern the workings of our most complex systems.