Representational Dissimilarity Matrix

How does the brain organize the flood of information it receives, transforming raw sensory input into coherent thoughts and concepts? Unlocking this neural code is a central goal of neuroscience, yet the sheer complexity of brain activity makes direct interpretation a monumental challenge. We cannot simply 'read' a concept like 'cat' from a pattern of firing neurons. This gap calls for a more abstract approach—one that focuses not on the implementation, but on the underlying structure of the representation itself. This article introduces the Representational Dissimilarity Matrix (RDM), an elegant method that provides precisely such a solution by capturing the 'geometry of thought.'

First, in ​​Principles and Mechanisms​​, we will explore the core theory behind RDMs, detailing how they are constructed from neural data and rigorously compared using the framework of Representational Similarity Analysis. Following this, the ​​Applications and Interdisciplinary Connections​​ chapter will demonstrate the remarkable power of this approach, showcasing how RDMs serve as a bridge to compare biological brains with artificial intelligence, map the high-speed dynamics of cognition, and even offer a new empirical lens on the age-old mystery of consciousness.

Imagine you want to understand how a friend's mind organizes the concept of "animals." You can't just plug a cable into their brain and download a map. But you could try something simpler. You could ask them a series of questions: "On a scale of 1 to 10, how different are a cat and a dog? How about a cat and a whale? A dolphin and a shark?" After you've gone through all the pairs, you would have a table—a matrix—where each entry represents the perceived "dissimilarity" between two animals. This table wouldn't tell you what a "cat" is in your friend's mind, but it would reveal the structure of their animal-concept space. You might find that mammals cluster together, or that predators are seen as similar regardless of their class. You have captured the geometry of their thoughts.

This is the central idea behind the ​​Representational Dissimilarity Matrix (RDM)​​, a wonderfully elegant tool that allows neuroscientists to take a snapshot of the geometry of representations in the brain, in computational models, or in behavior, and to compare these geometries in a rigorous, quantitative way.

At its heart, an RDM is simply a square, symmetric matrix where each entry, $d_{ij}$, records the dissimilarity between the brain's response to stimulus $i$ and stimulus $j$. The diagonal entries, $d_{ii}$, are always zero, as nothing is dissimilar from itself. This simple object has profound properties. It is an abstract description of a representational space, divorced from the nitty-gritty details of the space itself.

Consider two people whose brains represent animals in a similar way—cats are near dogs, whales are far from lizards. However, the specific populations of neurons encoding this information might be in completely different physical locations in their respective cortices. One person's "animal-space" might be a rotated or flipped version of the other's in the high-dimensional space of voxel activities. Yet, because a rigid rotation or reflection doesn't change the distances between points, their RDMs would be identical. The RDM captures the intrinsic, relational structure—the true representational geometry—while being invariant to the specific "implementation details" like the spatial layout of the neurons. It is the fingerprint of a representation.

To construct an RDM from brain data, we first need to measure the brain's response to a set of stimuli. Using a technique like fMRI, we can get a pattern of activation across many voxels (3D pixels of the brain scan) for each stimulus. Each pattern can be thought of as a single point in a high-dimensional "voxel space." Our task is to define a notion of "distance" or "dissimilarity" between these points.

There are several ways to do this, each with its own geometric intuition.

A common choice is the ​​Euclidean distance​​, the straight-line "ruler" distance between two points in voxel space. This is perhaps the most intuitive measure of how different two neural activation patterns are.

Another, more abstract and often more powerful, choice is the ​​correlation distance​​, defined as $d_{ij} = 1 - r(\mathbf{x}_i, \mathbf{x}_j)$, where $r$ is the Pearson correlation between the activation patterns for stimuli $i$ and $j$. This measure has a beautiful property: it is insensitive to the overall intensity, or "gain," of the neural responses. Imagine two stimuli that elicit response patterns with the exact same shape, but one is simply twice as strong as the other (e.g., the firing rates of all neurons are doubled). Their Euclidean distance would be large, but because their patterns are perfectly correlated, their correlation distance would be zero. This is often desirable, as it suggests the two stimuli are being encoded in a fundamentally similar way, differing only in response magnitude.

These different dissimilarity measures can, in some cases, be deeply related. For instance, if one first standardizes all the response patterns (a common preprocessing step called z-scoring), the squared Euclidean distance between them becomes perfectly proportional to the correlation distance computed on the original patterns. The constant of proportionality is simply $2(n-1)$, where $n$ is the number of features (voxels). This reveals a beautiful mathematical unity between two seemingly different ways of looking at the data, assuring us that for standardized data, both methods are capturing the same underlying geometry.

It's important to note that not all dissimilarity measures are created equal. Some, like the Euclidean distance or the ​​angular distance​​ between vectors, are true mathematical "metrics"—they obey properties like the triangle inequality ($d_{ik} \le d_{ij} + d_{jk}$). Others, like the popular correlation distance ($1-r$), do not always satisfy this, meaning they can't be perfectly represented in a simple Euclidean space without distortion. This is a subtle but important point, especially when we wish to visualize the geometry using techniques like Multidimensional Scaling (MDS).

Once we have our RDM—our "fingerprint" of a representation—we can do something remarkable: we can compare it to other fingerprints. This is the core of ​​Representational Similarity Analysis (RSA)​​. We might want to ask: does a specific layer in a deep neural network "see" the world in the same way as the human primary visual cortex? Or, does the brain's representation of faces switch from a feature-based geometry in early visual areas to a more categorical one in higher-level areas?.

To compare two RDMs, say one from the brain, $D^{(\text{brain})}$, and one from a computational model, $D^{(\text{model})}$, we first need to convert them into a format suitable for correlation. We do this by ​​vectorizing​​ the matrices—turning them into a long list of numbers. To do this correctly, we take all the unique dissimilarity values, which lie in the upper (or lower) triangle of the matrix, and line them up in a consistent order. We must exclude the diagonal zeros and avoid double-counting the symmetric entries, as including these would artificially and incorrectly bias our comparison. This gives us two vectors, $\mathbf{d}^{(\text{brain})}$ and $\mathbf{d}^{(\text{model})}$.

Now, how to compare them? We could use a standard Pearson correlation, but we face a challenge. The units of our two RDMs might be totally different—one might be based on fMRI BOLD signal changes and the other on unit activations in a computer model. A simple linear correlation might fail. The solution is to use ​​Spearman's rank correlation​​. This ingenious method first converts all the values in each vector into their ranks (1st, 2nd, 3rd, etc.) and then computes the Pearson correlation on these ranks. The result is a measure that only cares about the ordering of the dissimilarities. As long as both the brain and the model agree that the pair (cat, dog) is less dissimilar than (cat, whale), the rank correlation will be high, regardless of the absolute numerical values. This robustness to any monotonic (order-preserving) rescaling is what makes RSA so powerful, allowing us to compare the representational geometries of vastly different systems.

Finding a high correlation between a brain RDM and a model RDM is exciting, but a good scientist must be skeptical. How do we know our result is real and not just a fluke? And how do we know it's not driven by a trivial, uninteresting factor? RSA comes with a toolkit for just these questions.

To test for statistical significance, we use a ​​permutation test​​. Under the null hypothesis that there is no true relationship between the brain's and the model's representational structure, the stimulus labels are arbitrary. We can simulate this "null world" by randomly shuffling the labels of one of the RDMs (which corresponds to simultaneously permuting its rows and columns) and re-computing the correlation with the other, unshuffled RDM. By repeating this thousands of times, we build a distribution of correlations that could have occurred by pure chance. If our originally observed correlation is an extreme outlier in this distribution, we can be confident that our finding is statistically significant.

Next, we must worry about ​​confounds​​. Suppose we are comparing RDMs for a set of natural images. It's possible that both the brain and the model are simply sensitive to low-level properties, like the fact that two images have very similar pixel brightness or a similar amount of high-frequency detail. This could create a correlation that we might mistake for a shared understanding of higher-level content. To solve this, we can construct ​​confound RDMs​​ based on these low-level features (e.g., by computing dissimilarities between the images' pixel vectors or their spatial frequency spectra). We can then use multiple regression to ask whether our model of interest explains variance in the brain RDM above and beyond what can be explained by the confounds.

Finally, once we have a correlation that we believe is real and non-trivial, we need to know: is it a good correlation? The brain's signals are noisy. Even a perfect model of the "true" underlying representation would not correlate perfectly with our noisy data. The ​​noise ceiling​​ provides a benchmark for what is achievable. The ​​lower noise ceiling​​ is an estimate of how well we can predict one subject's RDM from the average of the others, giving us a baseline for a model with genuine predictive power. The ​​upper noise ceiling​​ estimates the correlation of the "true" (but unknown) group RDM with the noisy data, providing a theoretical upper limit for any model. By comparing our model's performance to these ceilings, we can judge whether it is approaching the limits of what the data can tell us.

Through this series of principled steps—from constructing an abstract geometric object from raw data to comparing it robustly and testing it rigorously—the Representational Dissimilarity Matrix provides a powerful and elegant framework for peering into the very structure of thought.

Now that we have acquainted ourselves with the principles of the Representational Dissimilarity Matrix (RDM), we can embark on a far more exciting journey. The real beauty of a great scientific tool lies not in its internal mechanics, but in the new worlds it allows us to explore. The RDM is more than just a matrix; it is a conceptual bridge, a kind of Rosetta Stone that lets us compare the "thoughts" of vastly different systems: biological brains, artificial minds, and even abstract theoretical models. By focusing on the pure geometry of relationships, the RDM allows us to ask profound questions about how information is structured and transformed across these different domains. Let us now explore some of the remarkable places this journey can take us.

One of the most thrilling pursuits in modern science is the quest to understand our own intelligence by building artificial versions of it. For decades, computer scientists have been developing artificial neural networks, such as Deep Convolutional Neural Networks (CNNs), that can "see" and categorize images with startling accuracy. But do these machines see the world like we do?

This is not a philosophical question; it is an empirical one that RDMs are perfectly suited to answer. Imagine we take a set of images—pictures of faces, cars, animals, and tools—and show them to both a human (whose brain activity we measure with fMRI) and a CNN. For a specific region of the human visual system, say the inferior temporal cortex where object recognition is thought to happen, we can compute a neural RDM. This matrix captures the brain's own "similarity space" for these objects. At the same time, we can feed the very same images into the CNN and compute a model RDM from the activation patterns of one of its layers.

If the machine has learned to represent the world in a way that is similar to the brain, then the geometries of their RDMs should match. A cat that the brain sees as "dissimilar" to a car should also be "dissimilar" in the CNN's representation. We can test this directly by simply correlating the entries of the two RDMs. If the correlation is high, it provides powerful evidence that the computational principles of the artificial layer may be capturing something true about the function of that brain region. This approach has revolutionized computational neuroscience, transforming the vague goal of "building a model of the brain" into a concrete, testable research program.

A high correlation between a model and the brain is an exhilarating start, but a good scientist is always a skeptic. You might ask, "What if the model is right, but for the wrong reason?" For instance, a model RDM based on abstract object categories (e.g., 'faces' are similar to each other, 'tools' are similar to each other, but faces and tools are dissimilar) might match a brain RDM. But what if the images of faces just happen to have more curves, and the images of tools more straight lines? The brain region might simply be responding to these low-level visual features, and our "category" model's success would be a complete illusion—a confound.

This is where the RDM framework reveals its statistical sophistication. Instead of just one model, we can build several. We construct our main hypothesis RDM (e.g., the category model) and then we construct other RDMs for all the alternative hypotheses and confounds we can think of—a low-level model based on pixel similarity, a shape model, a color model, and so on. We can then use statistical techniques like multiple regression or partial correlation to ask a much sharper question: "Does our category model explain unique variance in the neural RDM, even after we have accounted for all the variance explained by the low-level confound models?". This allows us to dissect the representation and isolate the specific features that a brain region truly encodes. We can even enter a whole suite of competing scientific theories, each expressed as an RDM, and let them fight it out, with the brain data as the ultimate judge, all while using rigorous statistical tests to control for the fact that we are making multiple comparisons.

While fMRI gives us beautiful maps of where representations exist in the brain, its timescale is sluggish. The actual process of seeing and recognizing an object is a lightning-fast cascade of computation, unfolding over just a few hundred milliseconds. To capture this, we need a technique with finer temporal resolution, like Magnetoencephalography (MEG) or Electroencephalography (EEG).

Applying the RDM framework here opens up a whole new dimension: time. Instead of one RDM for a whole fMRI scan, we can compute a separate neural RDM for every single millisecond of brain activity following the presentation of a stimulus. This gives us a "movie" of the brain's representational geometry as it evolves from moment to moment. We can then take a theoretical model RDM—say, one based on simple line orientations—and see when it correlates with the evolving neural movie. We might find it correlates at 60 milliseconds. Another model based on object category might only show a strong correlation starting at 150 milliseconds. In this way, we can trace the precise time course of information processing in the brain.

But we can do something even more elegant. Instead of comparing the brain's geometry to a fixed model, we can compare it to itself at different points in time. We can ask: "How similar is the brain's representational geometry at 100 milliseconds to its geometry at 110, 120, or 300 milliseconds?" By correlating the RDM from every time point $t_1$ with the RDM from every other time point $t_2$, we can construct a beautiful map called a ​​Temporal Generalization Matrix​​ (TGM).

The patterns in this map are incredibly revealing. If the geometry is only ever similar to itself at the exact same moment (a bright line along the diagonal of the TGM), it indicates a dynamic, feedforward cascade of processing, where the neural code is constantly being transformed. If, however, we see a square block of high correlation off the diagonal, it tells us that a specific representational geometry emerged and then remained stable for a period of time. This is a potential signature of a memory trace, a stable thought, or recurrent processing in a neural circuit. The TGM gives us a visual fingerprint of the brain's underlying computational dynamics.

A major headache for neuroscientists is that every person's brain is unique, not just in its tiny anatomical wrinkles but also in how its functional areas are laid out. The pattern of voxels in my brain that represents a "cat" is completely different from the pattern in yours. If we are looking for universal principles of the mind, how can we ever compare brain activity across different people?

The RDM already provides a partial solution. By abstracting away from the raw activity patterns to the relational geometry, we move to a space where comparisons are more meaningful. But we can go a step further. An ingenious technique called ​​hyperalignment​​ uses the brain's responses to a shared experience—like watching the same movie—to create a "functional" alignment across brains.

The method is a bit like finding a universal translator. It computes a transformation for each person's brain that rotates their high-dimensional neural activity space into a common space shared by everyone. It does this by finding the rotations that best align the trajectories of neural activity as subjects experience the same stimulus. Crucially, it doesn't need to know anything about the movie's content; it only uses the fact that the brains are responding to the same input stream. After hyperalignment, the idiosyncratic "noise" of each individual brain is largely filtered out, and the shared, stimulus-driven signal is amplified. When we then compute RDMs within this common space, we find that the similarity between subjects' representational geometries increases dramatically, revealing a universal code that was previously hidden.

So far, we have mostly treated brain regions as isolated islands of computation. But of course, the brain is a massively interconnected network. How do different regions coordinate their representations? This leads to the concept of ​​representational connectivity​​. The idea is as simple as it is powerful: we build a graph where the nodes are different brain regions. The weight of the edge connecting any two regions is defined as the similarity between their respective RDMs.

A strong edge between, say, the auditory cortex and a motor planning area doesn't just mean their activity levels rise and fall together. It means they share a common representational geometry—that they organize the experimental conditions (perhaps different spoken words) according to a similar relational structure. This is a much deeper and more meaningful form of connectivity. Once we have this network, we can apply all the powerful tools of graph theory to find "communities" of brain regions that operate on a shared code, or "hubs" that might integrate information from different representational formats. This elevates RSA from a tool for studying single regions to a true systems-level approach for understanding the brain's distributed architecture.

Perhaps the most profound application of the RDM is in the scientific study of consciousness itself. What is the difference in the brain between seeing something consciously and processing it unconsciously? It's a question that has puzzled philosophers and scientists for centuries.

Imagine an experiment where stimuli are flashed so briefly that on some trials a participant reports seeing them clearly, while on others they report seeing nothing at all. We can use RSA to look at the representational geometry in both cases. A fascinating hypothesis is that the information might be physically present in the brain's activity in both conscious and unconscious trials, but it is only organized into a meaningful, abstract structure when it enters our awareness.

Using the methods we've discussed, we can create separate RDMs for conscious and unconscious trials. Crucially, we must use advanced techniques—like cross-validated distances and noise ceilings—to ensure we aren't fooled by simple differences in signal strength. When we then test how well a high-level categorical model (e.g., "animal" vs. "object") explains these RDMs, a remarkable pattern can emerge. The category model might show a strong fit to the RDM from conscious trials, but no fit whatsoever to the RDM from unconscious trials. This would imply that consciousness is not merely the presence of information, but its organization into a stable, abstract, and useful geometry. With the RDM, we gain an empirical handle on one of science's deepest mysteries.

From testing artificial intelligence to mapping the dynamics of thought and even probing the nature of awareness, the Representational Dissimilarity Matrix has proven to be an exceptionally versatile and powerful idea. It teaches us that to understand a complex system, we should look not just at its parts, but at the beautiful and intricate geometry of their relationships.