SciencePediaHow does the brain transform sensory input into meaningful thought? Scientists theorize it does so by creating complex patterns of neural activity, where each concept has a unique "representational geometry." But comparing these intricate patterns—within the brain or between a brain and an AI—poses a significant challenge. Representational Similarity Analysis (RSA) offers a powerful solution, providing a common currency to measure and compare the shape of information across different systems. This article demystifies RSA, guiding you through its core principles and diverse applications. The first chapter, "Principles and Mechanisms," will unpack the foundational concepts of RSA, explaining how Representational Dissimilarity Matrices (RDMs) are constructed and compared. Following this, "Applications and Interdisciplinary Connections" will showcase how RSA is used to bridge the gap between brains and computational models, test major cognitive theories, and even probe the frontiers of consciousness and artificial intelligence.
How does the brain—that three-pound universe of electrified jelly—turn the photons bouncing off a face into the recognition of a friend? How does it distinguish the aroma of coffee from the scent of a rose? Scientists believe the brain does this by representing the world in patterns of neural activity. Each thought, each percept, each memory corresponds to a specific chord of neurons firing. If we could see these patterns, we might discover that the patterns for "cat" and "dog" are more similar to each other than either is to the pattern for "car." In other words, these internal representations have a shape, a geometry. Representational Similarity Analysis (RSA) is a powerful conceptual and mathematical framework that allows us to discover and compare these hidden geometries. It's like learning to read the mind's own map of the world.
The core tool of RSA is beautifully simple: the Representational Dissimilarity Matrix (RDM). Imagine you show a person a small set of images: a cat, a dog, a car, and a bicycle. For each image, you measure the pattern of activity across a few thousand voxels in their visual cortex using fMRI. The RDM is simply a square table that records how dissimilar the brain's activity pattern is for every possible pair of images.
The RDM is a "fingerprint" of the representation. It doesn't care about the specific activity of any single neuron or voxel. It only cares about the relationships between the patterns. A low number in the RDM for the cat-dog pair means their neural patterns are similar. A high number for the cat-car pair means they are very different. The matrix is symmetric—the dissimilarity from cat to dog is the same as from dog to cat—and the diagonal is always zero, as the dissimilarity of any pattern with itself is zero.
But how do we calculate this "dissimilarity"? The choice of metric is a crucial step that defines what aspect of the geometry we are measuring.
One straightforward choice is the Euclidean distance. If you imagine the activity of three voxels as a point in 3D space, the Euclidean distance is just the straight-line distance between the points for two different stimuli. This metric is intuitive, but it's sensitive to both the overall strength (magnitude) of the neural response and its specific pattern.
A more subtle and often more powerful choice is the correlation distance, defined as , where is the Pearson correlation between the activity patterns and . To grasp the intuition, imagine two neural activity patterns. One is and the other is twice as strong, . The Euclidean distance between them would be large. However, their correlation is a perfect , so their correlation distance is . This metric ignores the overall brightness or volume of the neural response and focuses purely on its shape or pattern. It asks, "Are these two patterns, regardless of their overall strength, playing the same 'melody' of relative activity across the neurons?" For this reason, correlation distance is a workhorse of RSA, allowing us to focus on the abstract pattern of information.
Of course, real neural data is noisy. Advanced methods use clever cross-validation techniques to compute more robust, unbiased distances that account for the structure of the noise, ensuring the RDM we build is as accurate a fingerprint as possible.
Here is where the magic begins. The RDM is an abstraction. It's a common format, a universal currency for representing geometry. This means we can create an RDM not just for a brain region, but for anything that produces a representation of our stimuli.
We can build an RDM from:
Now we have two RDMs—say, one from the brain and one from a deep neural network model. How do we compare their fingerprints? We can't just subtract their entries, because the dissimilarity values might be on completely different scales (e.g., fMRI values from to , model values from to ). The solution is to compare not the values themselves, but their ranks.
This is done using Spearman's rank correlation. We take all the unique dissimilarity values from the upper triangle of the brain RDM and rank them from smallest to largest. We do the same for the model RDM. Then, we calculate the standard Pearson correlation between these two lists of ranks. A high Spearman correlation means that pairs of stimuli that are similar in the brain are also similar in the model. It reveals a shared "what's-more-similar-to-what" structure, abstracting away the specific units or scales. This focus on rank order is what gives RSA its flexibility and power.
This power also defines its limits. Because rank-based RSA ignores absolute magnitudes, it cannot distinguish between geometries that are related by any order-preserving (monotonic) transformation. For example, a geometry based on Euclidean distance () and one based on squared Euclidean distance () will produce the exact same rank ordering and be perfectly correlated. Rank-based RSA tells us about the topology and ordering of the representational space, but not its absolute metric properties—a crucial distinction for interpreting results.
Let's see how this plays out in a hypothetical experiment inspired by real neuroscience. A research team studies vision using both fMRI, which has high spatial resolution, and MEG, which has high temporal resolution. They find that the RDM from the fMRI data in a high-level visual area has a near-perfect correlation with a detailed model based on complex visual features. However, the RDM from the MEG data, captured just milliseconds after the image appeared, correlates best with a much simpler model that only distinguishes between major categories (e.g., faces vs. objects). This is a beautiful result! It suggests a dynamic process: the brain first makes a rapid, coarse categorical judgment, and then, over the next few hundred milliseconds, fills in the rich, feature-based detail. RSA allows us to chart this unfolding of a thought.
However, great power comes with great responsibility. What if the categories in the simple model (faces vs. objects) also happen to differ in some low-level way, like their average pixel brightness or spatial frequency? A correlation with the category model might just be picking up on this simple visual confound, leading to a "reverse inference" fallacy where we wrongly conclude the brain is processing semantics when it's just processing pixels.
To make a stronger claim, a scientist must act as a detective, ruling out alternative explanations. This is done by building RDMs for the potential confounds—one for pixel similarity, one for spatial frequency, and so on. Then, using multiple regression, we can ask: "Does our semantic model explain variance in the brain's RDM above and beyond what these low-level confound models can explain?". Only by showing that a model makes a unique contribution can we make a robust scientific claim. This process of competitive model testing is at the heart of modern science, and RSA provides the perfect arena for it.
The beauty of RSA extends even further, connecting the descriptive world of geometry to the predictive world of decoding. A linear decoder is an algorithm that learns to "read out" information from a pattern of activity—for instance, deciding whether a pattern corresponds to a cat or a dog.
Imagine a brain region and a computational model have RDMs that are perfectly correlated after accounting for noise. This means their internal geometries are, up to a scaling and rotation, identical. This is a profound statement. It implies that the "code" for information is the same in both systems. Consequently, a linear decoder trained to read information from the model's representation can be directly transferred to the brain's representation with a simple linear transformation of its weights, and it will work perfectly.
This unifies two major approaches in computational neuroscience. Characterizing the representational geometry (RSA) and training a model to read out information (decoding) are two sides of the same coin. A well-understood geometry implies a predictable code. In this elegant connection, we see the true power of RSA: it is not just a tool for making pictures of the mind's contents, but a deep framework for understanding the very principles of neural computation.
Having grasped the principles of Representational Similarity Analysis (RSA), we now embark on a journey to see it in action. If the previous chapter was about learning the grammar of RSA, this chapter is about reading its poetry. RSA is far more than a dry statistical method; it is a conceptual lens, a veritable Rosetta Stone that allows us to translate and compare the language of information processing across brains, minds, and machines. Its power lies in abstracting away from the messy, specific details—the particular firing of a neuron, the activation of a single voxel, or the value of a node in an artificial network—to reveal the underlying shape or geometry of the information itself. Let us explore how this elegant idea unlocks answers to some of the most profound questions across the sciences.
Perhaps the most natural and impactful application of RSA is in computational neuroscience, where it serves as a crucial bridge between biological brains and the computational models designed to mimic them. For decades, a central goal has been to create models of brain functions like vision. But how do we know if a model is truly "good"? It's not enough for it to simply achieve the same outcome, like correctly identifying an object in a picture. We want to know if it solves the problem in the same way the brain does.
Imagine a deep convolutional neural network (CNN), a powerful computer vision model, and the primate's ventral visual stream, the series of brain regions responsible for object recognition. We can show both the CNN and a monkey the same set of images—a car, a face, a chair. RSA gives us a brilliant way to compare their internal workings. For each system, we construct a Representational Dissimilarity Matrix (RDM), an table summarizing how different the internal pattern of activity is for every pair of images. If the pattern for "car" is very different from "face" but similar to "truck" in the brain, a good model should exhibit a similar geometric relationship. By simply correlating the RDMs from the brain and the model, we can get a single, powerful number that tells us how well the model's representational geometry matches the brain's.
This approach allows for remarkable specificity. The brain is not one monolithic computer; it has specialized pathways. The ventral stream is the "what" pathway, identifying objects, while the dorsal stream is the "where" or "how" pathway, processing motion and spatial relationships. We can design models that are hypothesized to mimic one pathway or the other. For instance, we can create a model specializing in object shape and another specializing in motion. Using RSA, we can test these models against neural recordings from the corresponding brain areas—the Inferior Temporal (IT) cortex for form and the Middle Temporal (MT) area for motion. We would expect, and can verify, that the form model's RDM correlates highly with the IT's RDM but poorly with MT's, and vice versa for the motion model. This provides powerful evidence for functional specialization and validates our understanding of the brain's architecture.
Beyond validating models, RSA can be used to test grand theories of cognition. Consider the puzzle of memory. The Complementary Learning Systems (CLS) theory posits that we have two interacting memory systems: the hippocampus, which learns new episodes quickly and keeps them distinct (a process called pattern separation), and the neocortex, which slowly learns the general statistical structure of the world through consolidation. RSA provides a perfect tool to test this. We can show participants a set of items from different categories and measure brain activity immediately and again after a delay (e.g., after sleep).
CLS theory makes a clear prediction about the geometry of these representations. Immediately after learning, the hippocampus should treat every item as unique, creating highly distinct patterns even for items in the same category. Its RDM should fit a model where every item is different from every other. The neocortex, being a slow learner, will not have changed much. However, after consolidation, the neocortex should have extracted the shared information. Now, its representations for items in the same category (e.g., two different birds) should become more similar. The neocortical RDM should now show a strong correlation with a model based on semantic categories. By comparing RDMs from both brain regions at both time points to theoretical models of episodic and categorical structure, researchers can observe this beautiful handover of information, just as the theory predicts.
RSA also allows us to visualize the process of learning itself. When the brain learns, it refines its neural representations. Imagine training someone to become an expert at identifying different faces. Using RSA, we can track the changes in their brain's representational space. Before training, the patterns for different faces might be somewhat scattered. After training, as the person learns the key distinguishing features, the neural patterns for all faces become more similar to each other and more distinct from non-face objects. This manifests as an increase in the within-category similarity, or a decrease in the within-category dissimilarity. The points representing "faces" in the high-dimensional neural space cluster together more tightly. RSA makes this geometric tightening, a physical signature of learning and expertise, directly observable.
Armed with this powerful tool, scientists can venture into territories that were once the exclusive domain of philosophy. What is consciousness? What is the relationship between a dream and reality? Where do our cognitive abilities come from?
RSA provides a new way to hunt for the neural correlates of consciousness. Instead of just looking for brain areas that "light up" when a person is aware of something, we can ask a more sophisticated question: Does the brain's representation of the world become more structured and meaningful when we are conscious of it? We can present stimuli so briefly that they are sometimes consciously perceived and sometimes not. RSA allows us to test if a clear categorical structure—for instance, the separation of "faces" from "tools"—only emerges in the neural patterns on trials where the participant reports being aware of the stimulus. This shifts the search from a simple on/off switch to the emergence of informational geometry, providing a much richer picture of what consciousness does to neural processing.
The investigation can be pushed even further, into the very nature of internal experience. Think about three distinct conscious states: perceiving an apple, vividly imagining an apple, and dreaming of an apple. Are these experiences supported by a common neural code? Do they share a representational geometry? RSA allows us to frame this question precisely. We can construct an RDM from brain activity during perception, another during cued imagery, and—with clever techniques involving dream reports—a third from REM sleep. We can then correlate these RDMs. A high correlation would suggest that the brain uses a shared "representational format" for external and internal worlds, a truly profound insight into the mechanics of the mind.
The lens of RSA can also be pointed backward in time, to explore the evolutionary origins of cognitive abilities. A classic problem in evolutionary biology is distinguishing homology (a trait inherited from a common ancestor) from analogy (a trait that evolved independently to serve a similar function, like the wings of a bird and a bat). Consider the ability to estimate numbers, or "numerosity," which is found in both primates and birds. The brain regions involved—the prefrontal cortex (PFC) in primates and the nidopallium caudolaterale (NCL) in birds—are known to be homologous, deriving from a common ancestral brain structure. But is the computation for numerosity also homologous? Or did birds and primates independently evolve this trick? RSA provides a key piece of the puzzle. By comparing the detailed representational geometry of number in the PFC and NCL across various tasks, we can test if they share a common computational algorithm. If the RDMs from the two species show a strong, detailed correspondence, it's powerful evidence that the neural algorithm itself was inherited from their common ancestor, representing a true cognitive homology.
Finally, RSA can turn a series of static snapshots into a fluid motion picture of thought. Brain processes unfold in time, often on a millisecond scale. By applying RSA to time-resolved data (from techniques like EEG or MEG), we can compute an RDM at each time point. We can then correlate these RDMs across time, producing a Temporal Generalization Matrix (TGM). This matrix reveals the dynamics of a neural representation. A TGM that is bright only along its diagonal means the neural code is changing rapidly, with each moment's geometry being unique. A solid, bright square indicates a stable, sustained representation. A pattern that is bright off-diagonal reveals that a representation present at one point in time re-emerges later, suggesting a reactivation or a cyclical process. This "cross-time" RSA allows us to watch a representation form, stabilize, transform, and decay, revealing the dynamic choreography of the thinking brain.
The true beauty and testament to RSA's fundamental nature is that its utility is not confined to the brain. Its logic applies to any system that processes information and has internal states. This has made it an invaluable tool in the field of artificial intelligence, allowing us to perform a kind of "AI neuroscience."
Consider the challenge of transfer learning in machine learning. We might train a complex Graph Neural Network (GNN) on a massive dataset to predict one property of materials, like their formation energy. We then want to adapt, or "fine-tune," this network for a new, related task, like predicting the diffusion barrier for ions in a battery, for which we have much less data. Which parts of the network should we retrain, and which parts contain general knowledge that should be kept "frozen"?
RSA provides a brilliant answer. For each layer of the GNN, we can extract its representations for a set of materials and compute an RDM. We can also create a target RDM from the known diffusion barrier values. By correlating each layer's RDM with the target RDM, we can measure how well that layer's geometry aligns with the structure of the new problem. We might find that the early layers of the pre-trained network already have a geometry that aligns beautifully with the diffusion barrier problem, even though they were never trained on it. This tells us these layers have learned fundamental principles of crystal structure and should be frozen. Conversely, later layers might show poor alignment, but their geometry might drastically improve after fine-tuning. This analysis tells engineers, layer by layer, where the useful, transferable knowledge resides, enabling them to build more efficient and effective AI systems.
From testing classic theories of language lateralization in the human brain to optimizing the architecture of artificial intelligence, the applications of Representational Similarity Analysis are as broad as they are deep. It is a unifying concept that empowers us to ask the same question—"What is the shape of your information?"—of a neuron, a cortical column, a brain hemisphere, a conscious mind, and a silicon chip. By focusing on the abstract geometry of relationships, RSA reveals a hidden unity in the principles of information processing, making it one of the most vital and exciting ideas in modern science.