Partial Least Squares

In an era of unprecedented data collection, scientists are often confronted with a paradoxical challenge: having so much data that traditional analytical methods break down. When datasets feature far more variables than samples and suffer from high multicollinearity—a common scenario in fields from spectroscopy to genomics—standard techniques like Multiple Linear Regression become unreliable. This knowledge gap calls for a more robust approach, one capable of finding the predictive signal hidden within the noise. This article introduces Partial Least Squares (PLS), a powerful statistical technique designed precisely for this purpose. In the sections that follow, we will first delve into its core ​​Principles and Mechanisms​​, exploring how PLS masterfully transforms complex, high-dimensional data into meaningful, predictive models. Subsequently, we will journey through its diverse ​​Applications and Interdisciplinary Connections​​, revealing how this versatile tool provides critical insights in fields ranging from analytical chemistry to evolutionary biology and drug design.

So, we have a powerful tool that can seemingly look at a complete mess of data and pull out a clean, predictive signal. It feels a bit like magic. But as is always the case in science, it’s not magic—it's just a profoundly clever idea. Our job now is to pry open the lid of this box called Partial Least Squares (PLS) and understand the beautiful machinery inside. We won’t get lost in the gears and levers of every last equation, but we will come to appreciate the elegant principles that make it tick.

Let's start with a very modern problem. We live in an age of data. In many scientific fields, like the spectroscopic analysis of a chemical mixture, we can easily measure thousands of variables for a single sample. Imagine you are an analytical chemist trying to determine the concentration of a drug in a pill. You shine a light through it and measure the absorbance at 1200 different wavelengths. You do this for 25 different pills with known concentrations to build a calibration model. Now you have a mountain of data: a $25 \times 1200$ matrix of predictors.

Instinctively, you might think, "Great! More data is better." You might try to use a classic statistical tool, ​​Multiple Linear Regression (MLR)​​, to find a relationship. The idea is simple: assume the concentration is a weighted sum of the absorbances at every single wavelength. But when you try this, the model implodes. The coefficients it calculates are nonsensically huge, swinging wildly if you change even one sample. The model is utterly useless for prediction. What went wrong?

This is a classic case of what we call the ​​curse of dimensionality​​, and it has two partners in crime. First, you have far more variables ($P=1200$) than you have samples ($N=25$). Mathematically, this means you're trying to solve a system of equations with infinitely many solutions. Second, your variables are not independent. The absorbance at one wavelength is almost identical to the absorbance at the wavelength right next to it. This property, called ​​multicollinearity​​, is the final nail in the coffin for MLR. The core of MLR involves a mathematical operation that is equivalent to inverting a matrix, and when your data is highly collinear and you have more variables than samples, this operation becomes unstable, like trying to balance a pyramid on its point.

So, here is the paradox: our powerful instruments have given us so much data that our traditional methods break. This is precisely where PLS enters the stage. It was invented to solve this exact problem.

The fundamental idea behind PLS is this: instead of using all 1200 of our original, weak, and correlated variables, what if we could alchemically create a handful of new, powerful, and uncorrelated "super-variables"? These new variables, which we call ​​latent variables​​, aren't measured directly. They are constructed as specific, weighted combinations of the original variables.

Let's look at a thought experiment to see how powerful this idea can be. Imagine a simple case where we measure a response, $\mathbf{y}$ (like our drug concentration), and we have just two predictor variables, $\mathbf{x}_1$ and $\mathbf{x}_2$ (absorbances at two wavelengths). Let's say that when we test each predictor individually, we find that both are very weakly correlated with our response. A simple linear regression using just $\mathbf{x}_1$ or just $\mathbf{x}_2$ would give a terrible model, with a predictive power, or $R^2$, of less than $0.04$. It seems like our predictors are almost useless.

But what if there's a hidden relationship? In a cleverly designed scenario based on real chemical effects, it's possible that while $\mathbf{x}_1$ and $\mathbf{x}_2$ are individually weak, their combination is incredibly strong. For instance, suppose the true relationship is hidden in the sum of the two variables. PLS is designed to discover this. It doesn't just look at the variables you give it; it looks for the best way to combine them. In a specific numerical example, while the individual variables gave an $R^2$ of about $0.038$, a one-component PLS model discovered the perfect linear combination, producing a new latent variable that was perfectly correlated with the response. The result? A perfect model with an $R^2$ of $1.0$. The improvement wasn't just a few percent; it was a factor of 26!.

This is the core magic of PLS. It acts like a master synthesizer, taking your cacophony of redundant, weak measurements and finding the hidden harmony—the one linear combination that truly matters for predicting the thing you care about. It turns a pile of lead into a nugget of gold.

So, how does PLS know which combination to pick? Out of all the infinite ways to mix our 1200 variables, how does it find the one that works? This brings us to the most important conceptual difference between PLS and its famous cousin, ​​Principal Component Analysis (PCA)​​.

Imagine you're looking at a large dataset, say, the measurements of a hundred different traits on a thousand dinosaur fossils. You want to simplify this massive table of numbers. You might use PCA. PCA's goal is to find the directions of the ​​maximum variance​​ in your data. It asks, "In which direction do these fossils differ the most?" The first principal component might be "size"—the combination of measurements that best separates a T-Rex from a Compsognathus. PCA is brilliant for exploring the structure within a single dataset, but notice that it does this without any external guidance. It's just describing the data's own shape.

Now, suppose you also have data on what these dinosaurs ate. You want to predict "diet" (the response, $\mathbf{Y}$) from the fossil measurements (the predictors, $\mathbf{X}$). This is a job for PLS. PLS does not start by asking, "What is the biggest source of variation in the fossil measurements?" It might be that the biggest source of variation is something totally unrelated to diet, like an artifact of how the fossils were preserved.

Instead, PLS asks a much more pointed question: "What linear combination of fossil measurements varies in a way that is most related to the variation in diet?" Its guiding principle is not variance, but ​​covariance​​. It seeks to maximize the covariance between a linear combination of predictors ($t = \mathbf{X}\mathbf{w}$) and the response variable(s) ($u = \mathbf{Y}\mathbf{c}$). It's always looking at both datasets simultaneously, seeking the shared story between them.

A beautiful hypothetical illustration clinches this point. Imagine a chemical system where your measured signal, $\mathbf{X}$, is dominated by a huge interfering substance, but the actual analyte you care about, $\mathbf{Y}$, contributes only a tiny part of the signal. Furthermore, the variation of the interferent is completely unrelated (orthogonal) to the variation of your analyte. A method like Principal Component Regression (PCR), which first does PCA on $\mathbf{X}$ and then regresses $\mathbf{Y}$ on the principal components, would be hopelessly lost. It would find the first component to be the massive interferent signal and try to use that to predict your analyte, leading to a terrible model. PLS, on the other hand, guided by covariance, would completely ignore the large but irrelevant interferent signal. It would successfully find the small, hidden direction in $\mathbf{X}$ that truly covaries with $\mathbf{Y}$, building a much more successful model.

Let's peek, just for a moment, at the mathematics that accomplishes this feat. The task of finding weight vectors $\mathbf{w}$ and $\mathbf{c}$ that maximize the covariance between the scores $\mathbf{t} = \mathbf{X}\mathbf{w}$ and $\mathbf{u} = \mathbf{Y}\mathbf{c}$ is a well-defined optimization problem. The expression we want to maximize is fundamentally $w^{\top} (\mathbf{X}^{\top} \mathbf{Y}) c$.

The elegant solution to this problem comes from a cornerstone of linear algebra: the ​​Singular Value Decomposition (SVD)​​. You can think of SVD as a master negotiator for matrices. When applied to the cross-product matrix $\mathbf{X}^{\top}\mathbf{Y}$, SVD breaks it down into three components: a set of left singular vectors (which give us the optimal weights $\mathbf{w}$ for $\mathbf{X}$), a set of right singular vectors (which give the weights $\mathbf{c}$ for $\mathbf{Y}$), and a set of singular values. The largest singular value is precisely the maximum covariance it was looking for. The corresponding singular vectors give us the recipe for the first, most important latent variable.

PLS then performs a clever trick. It "deflates" the matrices, essentially subtracting out the information that has just been explained by the first latent variable. Then, it repeats the whole process on the residuals—what's left over—to find a second latent variable that captures the next biggest chunk of covariance. It continues this iteratively, building a small set of powerful, orthogonal latent variables until adding more doesn't improve the model's predictive ability. This is why it's called "Partial" Least Squares: we typically use only a small, partial set of these powerful new components.

Building a model is one thing; trusting it and learning from it is another. PLS offers tools for both.

Once the model is built, we naturally want to ask: which of my original 1200 wavelengths were most important for the prediction? We can go back and look. One of the most common ways to do this is by calculating the ​​Variable Importance in Projection (VIP)​​ score for each original variable. A VIP score summarizes the influence of a single variable across all the extracted latent components. A common rule of thumb is that variables with a VIP score greater than 1 are considered important to the model. This allows us to connect the abstract latent variables back to our physical measurements, perhaps identifying the key spectral bands associated with moisture or a particular chemical bond.

It's also crucial to remember that PLS is not a magic wand that works on any data you throw at it. It is one tool in a larger analytical workflow. Real-world data is messy. Your spectrometer's lamp might drift, causing a baseline offset in your data. The little glass cuvettes holding your samples might have tiny variations in thickness, changing the optical pathlength. These effects can introduce additive and multiplicative noise that has nothing to do with the chemistry you're trying to measure. A robust analysis pipeline uses ​​preprocessing​​ steps—like applying spectral derivatives to remove baselines or normalization techniques like Standard Normal Variate (SNV) to correct for pathlength differences—before the data ever gets to the PLS algorithm. A skilled scientist uses these tools to clean and prepare the data, allowing PLS to focus on the real task of finding the covariance between chemistry and signal.

Finally, we must always maintain a healthy Feynman-esque skepticism. What if the world is more complicated than the linear relationships PLS assumes? In a fascinating, if hypothetical, scenario, imagine a chemical system where an unmeasured "quenching" agent reduces your analyte's signal non-linearly. To make matters worse, imagine this quencher's concentration is secretly correlated with your analyte's concentration. In such a case, even a PLS model will be systematically fooled. It will try to approximate the underlying non-linear curve with a straight line, leading to a model that is consistently wrong in a predictable way—a systematic ​​bias​​. The model will under-predict at some concentrations and over-predict at others. This is a powerful reminder that all models are simplifications of reality. Their power comes not from being perfectly true, but from being useful approximations, and it's our job as scientists to understand their assumptions and their limits.

In the last chapter, we took apart the engine of Partial Least Squares. We saw the gears and levers—the weights, scores, and loadings—and understood how they work together. Now, the real fun begins. We are going to take this remarkable machine out for a drive and see where it can take us. You will see that PLS is far more than a dry statistical algorithm; it is a lens, a new way of seeing, that allows us to find simple, beautiful patterns hidden in the bewildering complexity of the world. Our journey will take us from industrial chemistry to the frontiers of drug design, from the ecology of a forest floor to the deepest questions of causality in biology.

Let's begin in a place where PLS first made its name and remains an indispensable workhorse: the analytical chemistry lab. Imagine you are working in quality control. A tanker of industrial solvent arrives, supposed to be a pure isomer of xylene, but you suspect it's contaminated with its cousins, other xylene isomers, and ethylbenzene. You turn to your trusty near-infrared (NIR) spectrometer, a machine that shines light on the sample and records which wavelengths are absorbed. The resulting spectrum should be a unique "fingerprint" of the chemical.

The problem is, the fingerprints of these very similar molecules are practically smeared on top of one another. Looking at the height of a single peak, the classic approach, is useless; it's like trying to identify a person in a crowd by only looking at the top of their head. This is where PLS comes to the rescue. It doesn't just look at one peak; it looks at the entire pattern of hunches, bumps, and slopes across the whole spectrum. It learns the subtle, collective "shape" of each molecule's signal, even when it's buried under the noise of the others. By training the model on a few known mixtures, PLS can then look at the messy spectrum from your industrial solvent and tell you, with remarkable precision, the concentration of each component.

This "unmixing" superpower is not limited to organic solvents. The same challenge appears when analyzing minerals for valuable rare-earth elements. The light emitted by your target element, say Dysprosium, might be completely swamped by the dazzling, overwhelming light show put on by common elements like Iron and Calcium in the sample. Again, PLS can be trained to recognize the faint, unique signature of Dysprosium against a complex and shifting background, turning an impossible analytical task into a routine measurement.

And the concept of a "spectrum" is broader than just light. In electrochemistry, one might want to measure the concentration of the neurotransmitter Dopamine in a sample of artificial cerebrospinal fluid. The trouble is, Ascorbic Acid (Vitamin C) is often present and generates an electrical signal that severely overlaps with Dopamine's. By applying a PLS model to the voltammetry data—a plot of electrical current versus applied voltage—one can simultaneously quantify both substances with an accuracy that would be unimaginable by simply looking at the blended peaks. In all these cases, PLS acts as a computational prism, cleanly separating signals that nature has hopelessly entangled.

Now, let's leave the chemist's lab and venture into the even more complex worlds of biology and ecology. Here, we are often less concerned with "how much" of a substance is present and more interested in "how" and "why" a system works. We are looking for principles, not just numbers.

Consider a fundamental question in biology: what makes some genes produce vast quantities of protein while others produce only a trickle? We can measure various features of a gene's sequence—its "Codon Adaptation Index" ($CAI$), its "Effective Number of Codons" ($N_c$), and so on. These are our predictors. But they aren't independent clues; they are all correlated aspects of the gene's overall strategy. Here, we can use PLS to predict protein abundance from these features. But something more profound happens. The latent variables that PLS constructs are not just mathematical tricks to improve prediction. They begin to look like something real. The first latent variable might represent a holistic "high-expression strategy"—a combination of features that, together, mark a gene for high output. PLS moves beyond being a mere predictive tool and becomes an explanatory one, revealing the hidden logic in the data.

This shift from prediction to explanation is even clearer in ecology. Imagine studying the roots of different plant species. A plant faces fundamental trade-offs. Should it build long, thin, "cheap" roots to explore a large volume of soil quickly? Or should it build short, dense, "expensive" roots that live longer and are more durable? It can't do both. Ecologists call this the "root economics spectrum." By measuring a suite of traits—like specific root length (SRL), root tissue density (RTD), and root diameter—and relating them to a key function like the rate of nitrogen uptake, we can use PLS to find the major axes of trait covariation. Often, the very first latent variable that PLS extracts corresponds beautifully to this economic spectrum. The loadings—the recipe for building that latent variable—tell us exactly which traits are involved in the trade-off. A positive loading for SRL and a negative loading for RTD on the same component quantitatively confirms the "live-fast-die-young" versus "slow-and-steady" trade-off. This isn't prediction; it's discovery. PLS has become a tool for uncovering the fundamental principles of life.

In many modern scientific fields, we are drowning in data. But it's a specific kind of drowning. We often have an enormous number of variables, or features, for a very small number of samples. Think of medicinal chemistry, where researchers are trying to design a new drug. For a single potential drug molecule, we can use a computer to calculate its steric (size) and electrostatic (charge) properties at thousands of points on a 3D grid surrounding it. This gives us thousands of predictor variables. But synthesizing and testing a molecule for its biological activity is slow and expensive, so we might only have a few dozen molecules in our training set.

This is the dreaded $p \gg n$ problem: many more predictors ($p$) than samples ($n$). For traditional regression methods like Ordinary Least Squares, this is a fatal condition. With more variables than samples, there are infinite possible solutions, and the methods break down completely, unable to distinguish a real signal from random noise.

This is where PLS truly shines. It was built for this challenge. By focusing only on the variation in the thousands of predictors that is maximally correlated with the drug's activity, PLS elegantly sidesteps the curse of dimensionality. It carves out a small, manageable number of latent variables from the impossibly vast space of predictors. It finds the handful of "themes" in the data that matter, ignoring the rest.

And the result is not a black box. The coefficients of the PLS model can be mapped back onto the 3D grid around the molecule. This creates a literal map for the chemist, highlighting regions in space where a bulky group would be favorable to activity (a positive steric coefficient) and other regions where a positive charge would be detrimental (a negative electrostatic coefficient). PLS provides a data-driven blueprint for designing the next, more potent drug molecule.

So far, we have seen PLS model an asymmetric relationship: a block of predictors $X$ is used to predict an outcome $y$. But science is full of symmetric questions. We don't want to predict the head from the tail, we want to understand how the head and tail relate to each other. How does one complex system "talk" to another?

This question led to a beautiful and powerful extension called ​​two-block PLS​​. Here, we analyze the relationship between two sets of variables, $X$ and $Y$, to find the axes of maximum covariation between them. The method seeks a pair of directions—one in the $X$-space and one in the $Y$-space—such that the scores of the samples projected onto these directions are maximally correlated.

This is the tool of choice for fields like geometric morphometrics, where scientists study the evolution of biological shape. Imagine you have landmark data quantifying the head shape ($X$) and tail shape ($Y$) of hundreds of fish species. Two-block PLS will find the dominant pattern of coordinated shape change. For example, the first pair of latent variables might reveal a strong evolutionary trend where a longer, thinner head is consistently associated with a more forked, streamlined tail—a major axis of covariation reflecting adaptation for faster swimming.

This framework is not just descriptive; it is a powerful tool for testing deep functional hypotheses. Consider the skulls of cichlid fishes, a group famous for its incredible diversity of feeding strategies. Some are "biters," relying on powerful jaws, while others are "suction feeders," relying on rapid mouth expansion. These different functions should impose different patterns of "morphological integration," or statistical coupling, between the various bones of the skull. For a biter, one would expect the jaw bones to be tightly integrated with the braincase bones that anchor the powerful closing muscles. For a suction feeder, the bones involved in expanding the mouth cavity—the suspensorium, hyoid, and opercular series—should be tightly integrated. By partitioning the skull landmarks into these functional modules and running two-block PLS between them, we can quantitatively test these predictions. We can literally see the demands of function written in the patterns of statistical covariance, a striking confirmation of how evolution shapes form.

We have seen PLS predict, explain, and relate. We now arrive at the ultimate scientific question: can it reveal cause and effect? Can this statistical tool become a philosopher's stone, turning the lead of correlation into the gold of causation?

The honest answer is no—at least, not by itself. Naively applying PLS (or any regression model) to observational data and interpreting the results causally is a perilous path, fraught with biases and false conclusions. Correlation famously does not equal causation.

However, a tool does not need to be a complete solution to be an invaluable part of one. In the hands of a careful scientist, within a rigorous experimental design, PLS can play a crucial role in the modern machinery of causal inference.

Let's consider the profound biological concept of "canalization"—the idea that development is robust and can buffer against perturbations to produce a reliable outcome. A biologist might hypothesize that an environmental stressor perturbs two different molecular pathways inside an embryo, but does so in opposite directions, such that their effects on a final physical trait cancel each other out, leaving the trait unchanged.

Testing this requires separating and estimating the effects along these opposing causal pathways. A naive PLS regression on the molecular data would fail, as it would likely mix the opposing signals. However, within a more sophisticated framework like instrumental variable analysis or structural equation modeling, PLS finds a new and powerful role. If we have a randomized experimental stressor, we can use it as a starting point. PLS can be used in a first step to brilliantly distill high-dimensional measurements (like transcriptomic data) into robust scores for the latent molecular pathways. Crucially, this distillation is guided by the known exogenous causes (the experimental stress and genetic background), not the final outcome, avoiding circular reasoning. These well-behaved, PLS-derived scores can then be passed to a formal causal model (like two-stage least squares) that is capable of correctly estimating the separate causal effects.

In this role, PLS is not the causal engine itself, but it acts as a critical pre-processor, a data-refining machine that constructs the clean, high-quality inputs that a causal engine needs to run. It demonstrates the frontier of modern science, where deep theoretical questions are answered by a thoughtful synthesis of experimental design, biological insight, and powerful computational tools like PLS.

From the murky brew of the chemist to the intricate dance of evolving species, PLS has proven to be an instrument of remarkable versatility. Its enduring power lies in its core philosophy: in a world of overwhelming complexity, the path to understanding is to find the few, simple, latent dimensions along which the most important stories unfold.