Regularization by Denoising

Reconstructing a clear image or signal from noisy, incomplete data is a fundamental challenge across science and engineering. While classical methods seek a balance between fitting the data and enforcing simple, predefined properties, they often fall short when the underlying structure is complex. This creates a gap: how can we leverage our most advanced knowledge about what signals should look like—knowledge often encapsulated in powerful, state-of-the-art denoising algorithms—within a principled reconstruction framework? This article tackles this question by exploring the revolutionary concepts of Plug-and-Play (PnP) priors and Regularization by Denoising (RED), which fuse iterative optimization with the power of modern denoisers.

This article charts a course from foundational theory to expansive applications. In the first section, "Principles and Mechanisms," we will dissect the mathematical journey from classical MAP estimation to the development of PnP and RED, uncovering how a denoising step can replace a formal regularizer and exploring the theoretical implications of this swap. Subsequently, in "Applications and Interdisciplinary Connections," we will witness the remarkable impact of this paradigm, seeing how it reshapes fields from computational imaging and deep learning to graph theory and computational physics, revealing a unifying principle for inference across diverse domains.

To journey into the world of modern image reconstruction is to witness a beautiful dialogue between two fundamental ideas: what our measurements tell us, and what we already know. An image taken by a telescope or a medical scanner is never perfect. It is a blurred, noisy version of the truth. Our task is to reverse this process, to take the imperfect data and reconstruct the pristine original. The challenge is that a direct, naive reversal is doomed to fail, amplifying the noise into a meaningless mess. We must be smarter. We need to regularize.

Imagine we are trying to reconstruct an unknown image, which we can represent as a vector of pixel values, $\boldsymbol{x}$. Our measurement, $\boldsymbol{y}$, is related to the true image through a known process, modeled by a matrix $\boldsymbol{A}$, and corrupted by some noise, $\boldsymbol{w}$. The relationship is simple: $\boldsymbol{y} = \boldsymbol{A}\boldsymbol{x} + \boldsymbol{w}$.

The most obvious approach is to find an $\boldsymbol{x}$ that best fits the data—that is, to minimize the "data fidelity" term, typically the squared error $\|\boldsymbol{y} - \boldsymbol{A}\boldsymbol{x}\|_2^2$. But as we mentioned, this path leads to disaster, as the noise gets magnified. The solution lies in a profound principle from Bayesian statistics: the ​​Maximum a Posteriori (MAP)​​ estimate.

The MAP framework tells us that the best estimate for $\boldsymbol{x}$ is the one that maximizes the probability of $\boldsymbol{x}$ given the data $\boldsymbol{y}$. Bayes' rule elegantly breaks this down into two components: the likelihood of observing the data $\boldsymbol{y}$ given an image $\boldsymbol{x}$, and the prior probability of the image $\boldsymbol{x}$ itself. Maximizing the probability is equivalent to minimizing its negative logarithm. This gives us a beautiful objective function to minimize:

If we assume the noise $\boldsymbol{w}$ is Gaussian, the data fidelity term becomes the familiar least-squares error, $\frac{1}{2\sigma_w^2}\|\boldsymbol{y} - \boldsymbol{A}\boldsymbol{x}\|_2^2$, where $\sigma_w^2$ is the noise variance. The second term, the regularizer, is where the magic happens. It encodes our "prior" knowledge about what images are supposed to look like. For instance, we might believe that natural images are sparse or have smooth patches, and we can design a function $\phi(\boldsymbol{x})$ that penalizes images that don't have these properties. The MAP objective then takes the form:

Here, $\lambda$ is a parameter that balances our trust in the data versus our belief in the prior. Notice how the noise variance $\sigma_w^2$ and $\lambda$ work together. The solution depends on their product, $\lambda\sigma_w^2$. If the noise is high (large $\sigma_w^2$), or if our belief in the prior is strong (large $\lambda$), the algorithm will lean more heavily on the regularizer to clean up the image.

Having an objective function is one thing; solving it is another. When the regularizer $\phi(\boldsymbol{x})$ is complex, as it often is for realistic priors, direct minimization is difficult. This is where operator splitting algorithms like the ​​Alternating Direction Method of Multipliers (ADMM)​​ come to the rescue.

ADMM tackles the problem by breaking it into smaller, manageable pieces. It introduces a new variable $\boldsymbol{v}$ and reformulates the problem as minimizing $f(\boldsymbol{x}) + g(\boldsymbol{v})$ subject to the constraint $\boldsymbol{x}=\boldsymbol{v}$, where $f$ is the data term and $g$ is the regularizer. The algorithm then proceeds in three simple steps, repeated until convergence:

1. ​​x-update:​​ Update $\boldsymbol{x}$ to best fit the data, while staying close to the current $\boldsymbol{v}$.
2. ​​v-update:​​ Update $\boldsymbol{v}$ to be a "cleaned-up" version of the current $\boldsymbol{x}$.
3. ​​u-update:​​ Update a "dual" variable $\boldsymbol{u}$ that tracks the disagreement between $\boldsymbol{x}$ and $\boldsymbol{v}$.

The crucial step for our story is the ​​v-update​​. This step mathematically takes the form of a ​​proximal map​​. The proximal map of a function $g$ is defined as:

This looks complicated, but its intuition is simple and beautiful: find a new point $\boldsymbol{x}$ that is a compromise. It wants to be close to the input $\boldsymbol{z}$ (the second term), but it also wants to have a low value for the regularizer $g(\boldsymbol{x})$ (the first term). In other words, the proximal map takes a noisy input and produces a "cleaned-up" output that respects our prior. ​​A proximal map is a denoiser.​​

This is not just an analogy. For the classic total variation prior, the proximal map performs edge-preserving smoothing. For the L1-norm prior (promoting sparsity), the proximal map is the soft-thresholding operator. This realization is the bridge from classical optimization to a whole new world of possibilities.

The discovery that the core step of many optimization algorithms is essentially a denoiser led to a brilliant and audacious idea. If the proximal step is a denoiser, why not just replace it with any powerful, state-of-the-art denoiser we can find? This is the essence of ​​Plug-and-Play (PnP) priors​​.

Instead of being limited to regularizers $\phi(\boldsymbol{x})$ for which we can write down an explicit formula and derive a proximal map, we can take a black-box denoiser—perhaps a complex algorithm like BM3D, or a deep neural network trained on millions of images—and simply "plug it in" to the ADMM iteration in place of the proximal step.

This is incredibly liberating. It allows us to implicitly use the rich and complex priors learned by these powerful denoisers without ever having to write them down mathematically. But this freedom comes with a profound question: What problem is the algorithm actually solving now? Are we still performing MAP estimation?

The answer is, in general, no. The PnP algorithm may still converge to a good-looking image, but it's not necessarily the minimizer of the original MAP objective. The reason is that an arbitrary denoiser is not necessarily the proximal map of any well-behaved (proper, lower-semicontinuous, convex) function.

For an operator to be a true proximal map of a convex function, it must satisfy a strict mathematical property called ​​firm nonexpansiveness​​. This is a kind of stability condition, stronger than just being nonexpansive (i.e., not increasing distances). More deeply, it relates to a property called cyclic monotonicity, which guarantees the existence of an underlying convex "potential energy" function.

Many of the most powerful denoisers do not satisfy this property. Consider a simple linear denoiser that works by convolving the image with a non-symmetric filter. Its corresponding matrix operator will not be symmetric, and a non-symmetric linear operator cannot be the proximal map of a convex function. When we use such a denoiser, the PnP-ADMM algorithm is no longer descending on a single energy landscape. Instead, it is converging to what is known as a ​​consensus equilibrium​​: a point that simultaneously satisfies the data constraints and the "opinion" of the denoiser.

While PnP offers great practical power, its departure from a clear optimization objective can be unsettling for theorists. This is where ​​Regularization by Denoising (RED)​​ enters, attempting to restore a rigorous energy-based framework.

The philosophy of RED is to turn the problem on its head. Instead of starting with a regularizer and hoping its proximal map is a good denoiser, let's start with a good denoiser, $D(\boldsymbol{x})$, and use it to define a regularizer.

A key piece of inspiration comes from a beautiful statistical result known as ​​Tweedie's formula​​. For an important class of denoisers—namely, the Minimum Mean-Squared Error (MMSE) estimator for recovering a signal from Gaussian noise—there is an exact relationship between the denoiser and the underlying probability distribution of the data. Specifically, the "denoising residual," $\boldsymbol{z} - D(\boldsymbol{z})$, is proportional to the gradient of the log-probability of the data, $\nabla \log p(\boldsymbol{z})$.

This is a revelation! It tells us that the vector field defined by the denoising residual, $\boldsymbol{x} - D(\boldsymbol{x})$, is a ​​gradient field​​ (or a conservative field). In vector calculus, we learn that a vector field is a gradient field on a simple domain like $\mathbb{R}^n$ if and only if its Jacobian matrix is symmetric. This gives us a concrete condition on our denoiser: for the residual field to be integrable into a scalar potential energy, the denoiser's Jacobian, $J_D(\boldsymbol{x})$, must be symmetric. This is the cornerstone ​​integrability condition​​ of RED.

If this condition holds, we can assert the existence of a regularizer, $R(\boldsymbol{x})$, such that $\nabla R(\boldsymbol{x}) = \boldsymbol{x} - D(\boldsymbol{x})$. We can then confidently define our optimization problem as minimizing the data term plus this new regularizer, $\min_{\boldsymbol{x}} \{ f(\boldsymbol{x}) + \lambda R(\boldsymbol{x}) \}$, and solve it using standard methods like gradient descent, since we explicitly know the gradient of our regularizer.

Of course, subtleties remain. A common form for the RED regularizer is $R(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^\top(\boldsymbol{x} - D(\boldsymbol{x}))$. For the gradient of this specific functional to equal the simple residual $\boldsymbol{x} - D(\boldsymbol{x})$, the denoiser must satisfy not only Jacobian symmetry but also a homogeneity property. If these assumptions don't hold, the gradient expression becomes more complex. A principled way to ensure the integrability condition is to construct the denoiser from a scalar potential in the first place, for instance by defining $D(\boldsymbol{x}) \triangleq \boldsymbol{x} - \nabla s(\boldsymbol{x})$, which guarantees a symmetric Jacobian by construction.

To see the relationship between PnP and RED more clearly, let's consider a simple case where we know everything. Suppose our prior knowledge is captured by a simple quadratic potential, $\phi(\boldsymbol{x}) = \frac{1}{2} \boldsymbol{x}^{\top} \boldsymbol{L} \boldsymbol{x}$, where $\boldsymbol{L}$ is a symmetric matrix (e.g., a discrete Laplacian, which penalizes non-smoothness). This is the foundation of classical Tikhonov regularization.

The proximal map corresponding to this prior, which serves as our denoiser, is a linear filter given by $D_{\tau}(\boldsymbol{x}) = (\boldsymbol{I} + \tau \boldsymbol{L})^{-1} \boldsymbol{x}$, where $\tau$ controls the strength.

Now, let's see what PnP and RED do with this denoiser.

• If we plug this denoiser into the ​​PnP-ADMM​​ framework, the algorithm finds a fixed point that is the solution to a Tikhonov problem with an effective regularization matrix of $\Gamma_{\mathrm{PnP}} = \rho \tau \boldsymbol{L}$, where $\rho$ is the ADMM penalty parameter.
• If we use this same denoiser within the ​​RED​​ framework, we solve a Tikhonov problem with an effective regularization matrix of $\Gamma_{\mathrm{RED}} = \beta \tau \boldsymbol{L} (\boldsymbol{I} + \tau \boldsymbol{L})^{-1}$, where $\beta$ is the RED regularization parameter.

Notice that both methods recover a form of classical Tikhonov regularization, which confirms their sensibility. But crucially, the effective regularizers are different. This simple example beautifully crystallizes the distinct philosophies of PnP and RED. PnP's solution is shaped by the interplay of the denoiser and the dynamics of the splitting algorithm, while RED's solution is determined by the explicit construction of its regularizer from the denoiser.

At the end of the day, all these methods rely on a denoiser to inject prior knowledge. But what makes a denoiser "good," and how do we set its parameters, like its noise level $\sigma$? A simple statistical model provides a wonderfully clear intuition through the lens of the ​​bias-variance trade-off​​.

Imagine our estimator is just the denoiser applied to our noisy data, $\hat{\boldsymbol{x}}_\sigma = D_\sigma(\boldsymbol{y})$.

• If we ​​under-regularize​​ (choose a very small $\sigma$), the denoiser trusts the input data too much. The resulting estimate has low bias (it's accurate on average) but suffers from high variance (it's still very noisy). In the limit $\sigma \to 0$, the estimator simply returns the noisy data.
• If we ​​over-regularize​​ (choose a very large $\sigma$), the denoiser distrusts the data completely and relies almost entirely on the prior. The estimate has very low variance (it's very smooth and clean) but can have high bias (it might be smoothed so much that important features of the true signal are erased). In the limit $\sigma \to \infty$, the estimator might just return the average image (e.g., a black screen).

The optimal choice of $\sigma$ is the one that perfectly balances this trade-off to minimize the total error. In the idealized case of a Gaussian signal and Gaussian noise, the minimum error is achieved when the denoiser's noise parameter $\sigma$ is set to be exactly equal to the true noise level $\tau$ of the measurements. This provides a powerful guiding principle: the denoiser should be calibrated to the noise it is expected to remove. This simple idea connects the abstract operator theory and complex algorithms back to a fundamental and intuitive statistical principle, revealing the deep unity that underlies the art and science of seeing the unseen.

In our previous discussion, we uncovered the central principle of Regularization by Denoising (RED): the astonishingly simple yet powerful idea that any good denoiser can serve as a universal prior for solving inverse problems. The recipe is elegant: combine a data-fidelity term, which keeps our solution true to the measurements, with a pass through a denoiser, which ensures our solution looks "plausible."

This sounds beautiful in theory, but the true measure of a scientific idea is its power and reach. Where does this simple concept actually take us? What new doors does it open, and what old puzzles does it help solve? Prepare for a journey, because the answer extends far beyond cleaning up noisy images, reaching into the design of intelligent algorithms, the study of complex networks, and even the fundamental principles of computational physics.

Let's begin with the most natural home for RED: computational imaging. Scientists and engineers are constantly trying to see the invisible, whether it's mineral deposits miles underground or the chemical composition of a distant galaxy. The challenge is that we can rarely measure what we want directly; instead, we measure some scrambled, incomplete, and noisy version of it.

Consider the task of hyperspectral imaging, where we aim to capture an image across hundreds of different spectral bands. A single "data cube" can contain billions of values. Acquiring all this data can be prohibitively slow or expensive. But what if we don't have to? The theory of compressed sensing tells us that if a signal has some underlying structure, we can reconstruct it from far fewer measurements than we thought possible. For hyperspectral images, a key piece of structure is the correlation across spectral bands; the image often has a "low-rank" structure. Here, the RED framework shines. We can design a "denoiser" whose job isn't just to remove random noise, but to enforce this low-rank structure, effectively projecting any messy intermediate solution onto the space of plausible, structured hyperspectral images. By incorporating such a denoiser into our reconstruction algorithm, we can dramatically reduce the number of measurements needed, making previously impractical imaging technologies feasible.

This same principle applies across a vast array of imaging sciences. In computational geophysics, we try to map the Earth's subsurface from a handful of seismic or electromagnetic measurements. The underlying geology imposes structure on the solution—for example, it is often piecewise-constant. A denoiser trained to recognize and promote this geological structure can be "plugged into" a standard geophysical inversion algorithm, leading to clearer and more accurate subsurface maps. In each case, the story is the same: the denoiser acts as an expert consultant, telling the algorithm what a "reasonable" solution ought to look like based on prior knowledge of the domain.

Perhaps the most profound impact of the RED and Plug-and-Play (PnP) philosophy has been on the art of algorithm design itself. It has blurred the line between classical optimization and modern deep learning, creating a new class of hybrid, intelligent algorithms.

The idea is called ​​deep unrolling​​. Imagine a classic iterative algorithm, like the Alternating Direction Method of Multipliers (ADMM), which solves a problem by repeatedly performing a sequence of smaller, simpler steps. We can take this loop and "unroll" it, turning each iteration into a layer of a deep neural network. The mathematical operations of the original algorithm—like matrix multiplications—become fixed parts of the network architecture, while the "denoising" step, traditionally a fixed mathematical function, is replaced by a powerful, trainable denoiser like a Convolutional Neural Network (CNN).

The result is something remarkable: a deep network whose architecture is not arbitrary but is born from the principled structure of a proven optimization algorithm. We can then train this entire network from end to end, allowing the data to fine-tune not just the denoiser but other algorithmic parameters as well. We are no longer just using a neural network as a black box; we are building a network in the image of an algorithm.

But is this just a clever engineering trick, or is there deeper mathematical truth to it? Is the denoiser just a heuristic, or does it represent a true physical prior? Remarkably, for a broad class of denoisers, the RED framework can be placed on the solid ground of classical physics and statistics. If a denoiser is "conservative"—a mathematical condition meaning its output can be described as the gradient of some scalar potential function, $\Psi(x)$—then using it as a regularizer is equivalent to minimizing a classical energy function. The implicit regularizer takes the explicit form $R(x) = \frac{1}{2}\|x\|^2 - \Psi(x)$. This is a beautiful and reassuring result. It tells us that the new, data-driven approach of RED is deeply connected to the bedrock of Bayesian estimation and the principle of minimizing energy, which has guided physics for centuries.

The true power of a great idea is its generality. While RED was born in the world of images, its core principle—separating data fidelity from a structural prior—applies to any type of data, no matter how abstract.

Let's leave the world of pixels and venture into the world of networks. Imagine you are a sociologist trying to understand the community structure of a social network, but you can only poll a small, random fraction of the relationships. Can you reconstruct the full network and identify the communities? This is an inverse problem on a graph. The "signal" is the graph's adjacency matrix. The "prior" is the knowledge that the network is organized into densely connected communities with sparse connections between them. We can design a "graph denoiser" whose job is to take a noisy, incomplete graph and enforce this community structure—for instance, by shrinking the weights of edges that appear to span communities. Plugging this denoiser into a PnP-ADMM algorithm allows us to recover the hidden community structure from surprisingly little information.

We can go even deeper. When we use a modern Graph Neural Network (GNN) as our denoiser, what is it actually learning? It turns out that, under common conditions, the GNN is implicitly learning to penalize "high-frequency" components of the signal on the graph. A signal's "frequency" on a graph is measured by how rapidly it varies between connected nodes, a concept captured by the eigenvalues of the graph Laplacian operator. A GNN denoiser, in seeking to make the graph signal "smoother," ends up rediscovering a classical concept from spectral graph theory: the Sobolev semi-norm, which is precisely a penalty on high-frequency graph signals. Once again, a modern, data-driven method reveals its deep connection to a classic, principled mathematical idea.

A physicist learns that no theory is complete without understanding its subtleties, its failure modes, and its connections to other, seemingly unrelated phenomena. The same is true for RED.

One of the most elegant strategies for using RED is inspired by ​​homotopy methods​​ in mathematics—the art of solving a hard problem by starting with an easy one and slowly deforming it into the hard one. A RED reconstruction can be a difficult, non-convex optimization problem with many "spurious" solutions. We can make it easier by starting with a very strong denoiser (a large regularization parameter $\sigma$). This corresponds to a heavily smoothed, simplified problem that is easy to solve. We then run our PnP algorithm while gradually decreasing $\sigma$ in a pre-defined schedule. This "continuation" strategy allows the solution to track a path from the simple, blurry optimum toward the sharp, detailed solution of our target problem, avoiding pitfalls along the way. It is the algorithmic equivalent of annealing, gently guiding the system to its true ground state.

Furthermore, we must be careful experimentalists. A denoiser is a tool, and tools must be calibrated. A denoiser trained to remove noise of a certain variance, $\sigma_{\text{train}}^2$, may not perform optimally inside a PnP algorithm where the "effective" noise of the iterates is a different value, $\sigma_{\text{eff}}^2$. If the algorithm's internal noise is higher than the training noise, the denoiser will be too timid and will under-regularize the solution, leaving artifacts. If the internal noise is lower, the denoiser will be too aggressive and will over-regularize, blurring away fine details. The solution is an adaptive strategy: estimate the effective noise at each iteration and adjust the denoiser's strength on the fly. This creates a feedback loop that ensures the denoiser is always operating under the right conditions. A similar "annealing" strategy can be used to mitigate bias when the denoiser was trained on a different type of data than what it's being applied to, by gradually reducing the denoiser's influence as our solution gets closer to satisfying the measurements.

Finally, let us end with the most striking example of the unity of science. The nonlocal operators used in modern image processing are defined by integrals with a peculiar kind of "hypersingular" kernel, of the form $|\mathbf{x}-\mathbf{y}|^{-d-2s}$. This mathematical structure describes interactions that are both long-range and intensely strong at short distances. For decades, physicists and engineers in the field of ​​computational electromagnetics​​ have wrestled with exactly the same type of hypersingular integrals when calculating the fields generated by electric currents on antennas and other complex surfaces. The sophisticated mathematical and numerical techniques they developed—singularity subtraction, special-purpose quadrature rules, splitting the problem into "near-field" and "far-field" interactions—are not just analogous to what is needed in image processing; they are, in fact, the very same tools. The methods to calculate the radiation pattern of a stealth aircraft can be directly transferred to build better denoising algorithms for your camera.

From a simple idea about denoising, our journey has taken us through the theory of deep learning, the foundations of statistical inference, the analysis of complex networks, and the core of computational physics. Regularization by Denoising is far more than a clever trick; it is a profound and unifying principle that reveals the deep and often surprising connections that bind together the world of data, structure, and inference.