ResNet: From Residual Blocks to Dynamical Systems

Building ever-deeper neural networks has long been a central goal in artificial intelligence, promising greater expressive power and performance. However, this ambition was historically thwarted by a fundamental mathematical barrier: the vanishing and exploding gradient problem, which rendered the training of very deep networks unstable and ineffective. This article addresses this challenge by dissecting the revolutionary Residual Network (ResNet) architecture. First, under 'Principles and Mechanisms', we will explore how ResNet's simple yet elegant 'skip connections' transform the network's structure from a problematic chain of multiplications into a stable series of additions. Following this, the 'Applications and Interdisciplinary Connections' chapter will reveal the profound implications of this design, showing how ResNet is not merely an engineering trick but a reflection of fundamental principles found in dynamical systems, physics, and even biology. This journey will illuminate how a solution to a deep learning problem unveiled a unifying concept across science.

Imagine you are a peculiar architect tasked with building a skyscraper of immense, possibly infinite, height. You have a blueprint for a single, standard floor. But this blueprint has a flaw. Depending on the construction crew, each new floor might be slightly smaller or slightly larger than the one below it. If each floor is just 1% smaller, after 100 floors, the available space is only $(0.99)^{100} \approx 0.366$ of the original ground floor area. It shrinks into nothingness. If each floor is 1% larger, the space becomes $(1.01)^{100} \approx 2.7$ times the original. It explodes into instability. Your job seems impossible; you are balanced on a mathematical knife's edge.

This is precisely the predicament deep learning practitioners faced when trying to build very deep neural networks. Each layer of the network is a floor in our skyscraper. As information passes forward through the layers, or as gradients are passed backward during training, they are repeatedly multiplied by the transformations of each layer. The result? The signal either vanishes to zero or explodes to infinity. How do we build a stable skyscraper that reaches for the clouds? The answer, it turns out, is not in a more precise blueprint for the floor, but in adding a simple, elegant feature: an express elevator.

A traditional deep neural network is a function of functions. The input $x_0$ goes into the first layer to produce $x_1 = f_0(x_0)$. This output then becomes the input for the next layer, $x_2 = f_1(x_1)$, and so on. After $L$ layers, the final output is a deeply nested composition: $x_L = f_{L-1}(f_{L-2}(\dots f_0(x_0)\dots))$.

When we train such a network, we need to understand how a small change in an early parameter affects the final loss. The chain rule of calculus dictates that this dependency is found by multiplying the local derivatives (Jacobian matrices) of every single layer between the parameter and the loss: $J_{\text{total}} = \frac{\partial x_L}{\partial x_s} = J_{L-1} \cdot J_{L-2} \cdots J_s$ This long chain of matrix multiplications is the source of our skyscraper's instability.

Let's strip this down to its bare essence with a toy model, a network of just one neuron per layer. Here, the "matrices" are just scalars. Suppose each layer's transformation is, locally, just multiplication by a factor $a$. The output after $L$ layers is $x_L = a^L x_0$. The gradient of the output with respect to the input is simply $a^L$. If $|a| \lt 1$, say $a=0.9$, then after just 20 layers, the gradient is scaled by $0.9^{20} \approx 0.12$. It has almost vanished. If $|a| \gt 1$, say $a=1.1$, the gradient is scaled by $1.1^{20} \approx 6.7$. It's exploding. To maintain a stable signal, we would need $a$ to be almost exactly 1, a condition that is virtually impossible to maintain across a complex, learning network.

You might think that clever initialization schemes, like the celebrated ​​He initialization​​ designed specifically for modern activation functions, would solve this. And they help, immensely. They set the initial weights so that the variance of the signal is preserved, on average, from one layer to the next. Yet, even with this careful setup, the cumulative effect of randomness and nonlinearities in a very deep "plain" network inevitably pushes the system off this knife's edge. Empirical studies show that in a plain network of over 100 layers, the variance of the activations and the norm of the Jacobian still drift exponentially towards zero or infinity, crippling the training process. The tyranny of multiplication persists.

The breakthrough of the ​​Residual Network (ResNet)​​ was to change the game from multiplication to addition. Instead of hoping to learn a perfect identity transformation $x_{l+1} \approx x_l$, which is hard, a ResNet re-frames the problem. The core building block, the ​​residual block​​, is defined as: $x_{l+1} = x_l + \mathcal{F}(x_l)$ Here, $x_l$ is passed through directly via a ​​skip connection​​ (our "express elevator"), and the function $\mathcal{F}(x_l)$ learns the ​​residual​​—the part that needs to be added to $x_l$ to get the desired output.

The intuition is profound. If a layer is not useful, the network can easily learn to make $\mathcal{F}(x_l) \approx 0$ by driving its weights toward zero. In this case, $x_{l+1} \approx x_l$, and the signal passes through untouched. The network has a built-in "path of least resistance" for information to flow.

Let's see how this demolishes our vanishing gradient problem. In our simple scalar model, the update is now $x_{l+1} = x_l + ax_l = (1+a)x_l$. The gradient is now proportional to $(1+a)^L$. If $a$ is a small number (as we'd expect if the network is learning a small correction), say $a=0.01$, then we are multiplying by $1.01$ repeatedly. The gradient is stable. If $a=-0.01$, we multiply by $0.99$. Still stable.

The difference is not subtle; it is astronomical. For a network of depth $L=20$ with a layer transformation that shrinks the signal by half ($a=0.5$), the plain network's gradient is attenuated by $0.5^{20}$, which is less than one in a million. The residual network's gradient is amplified by $(1+0.5)^{20} \approx 3325$! The residual connection has turned a dead gradient into a vibrant signal.

This same logic holds in the full matrix-vector world. The Jacobian of the residual block is no longer just the Jacobian of the transformation, $J_{\mathcal{F}}$, but rather $I + J_{\mathcal{F}}$. What does this "plus I" do? It fundamentally changes the spectrum of the operator. If a vector $v$ is an eigenvector of $J_{\mathcal{F}}$ with eigenvalue $\lambda$, then it is also an eigenvector of $I + J_{\mathcal{F}}$, but with eigenvalue $1+\lambda$. All eigenvalues are shifted by one!

If the residual function $\mathcal{F}$ is learning a small correction, its Jacobian $J_{\mathcal{F}}$ will have eigenvalues close to zero. The eigenvalues of the full block Jacobian, $1+\lambda$, will therefore be clustered around 1. When we multiply these Jacobians together in the chain rule, we are multiplying matrices whose eigenvalues are all close to 1. The result is a total transformation that is stable, neither vanishing nor exploding. This is why a ResNet with hundreds or even thousands of layers can be trained effectively, while a plain network of the same depth fails completely.

The shift from multiplication to addition has an even deeper, more beautiful interpretation. Let's write the residual update rule with a small scaling factor $h$, representing the "strength" of the residual block: $x_{l+1} = x_l + h\mathcal{F}(x_l, l)$ Now, let's rearrange it slightly: $\frac{x_{l+1} - x_l}{h} = \mathcal{F}(x_l, l)$ If you've ever studied calculus or physics, this form should ring a bell. It is the spitting image of the ​​forward Euler method​​, a fundamental technique for finding the approximate solution to an ​​Ordinary Differential Equation (ODE)​​ of the form $\frac{dx}{dt} = \mathcal{F}(x, t)$.

This connection is transformative. A ResNet is not just a discrete stack of layers. ​​It is the discretization of a continuous process.​​ The input $x_0$ is the state of a system at time $t=0$. The network's layers do not perform arbitrary transformations; they compute the velocity vector $\frac{dx}{dt}$ that pushes the state along a continuous trajectory. The final output $x_L$ is simply the state of the system at some final time $T$.

This ODE perspective gives us a whole new language to understand deep learning.

• ​​Depth:​​ The depth $L$ of the network is simply the number of steps we take to solve the ODE. A "deeper" network can be seen as using a smaller step size $h$ for a more accurate approximation of the continuous trajectory.
• ​​Stability:​​ The rich field of numerical analysis for ODEs can be brought to bear on network design. For instance, the stability of the Euler method for the test equation $\dot{x}=\lambda x$ requires $|1+h\lambda| \lt 1$. This tells us that for a given system (defined by the eigenvalues $\lambda$ of $\mathcal{F}$'s Jacobian), there's a maximum "step size" $h$ we can use before our numerical simulation—our network—becomes unstable and explodes.

Viewing a ResNet as a continuous-time dynamical system unifies the discrete world of layers with the continuous world of calculus, revealing an unexpected elegance and structure behind the architecture.

Is the ResNet a perfect, infallible architecture? Not quite. The elegant mechanism of shifting eigenvalues close to 1 is a powerful heuristic, not an ironclad guarantee. What happens if, during training, the eigenvalues of the Jacobians $I + J_{\mathcal{F}}$ are consistently slightly greater than 1? For instance, if the spectral norm $\|I + J_{\mathcal{F}}\|_2 \ge 1+\gamma$ for many layers, the product of these norms can still grow exponentially, leading to the dreaded ​​exploding gradient problem​​.

But once again, our theoretical understanding can guide us to a more robust design. Consider a "scaled residual" architecture: $x_{l+1} = (1-\beta)x_l + \alpha \mathcal{F}(x_l)$ Here, $\beta \in (0,1)$ acts like a damping term on the identity path, and $\alpha$ scales the residual. We can now ask: how can we choose $\alpha$ and $\beta$ to guarantee stability? By analyzing the spectral norm of this new Jacobian, we can derive a precise condition. To ensure the norm never exceeds 1, we must have $\alpha \le \frac{\beta}{L}$, where $L$ is the Lipschitz constant of the residual function $\mathcal{F}$. This provides a principled way to design architectures that are provably stable against explosion. A similar analysis can provide a bound on the weights themselves to ensure forward signal stability.

Finally, the additive nature of ResNets gives them another remarkable property. Because the final output is effectively a long sum of transformations ($x_L = x_0 + \sum \mathcal{F}_l(x_l)$), ResNets behave as if they are an ​​ensemble​​ of many shallower networks. Dropping a block during training or testing doesn't break the network; it simply removes one term from the sum. This makes the network remarkably robust. This structure also creates a multitude of paths of varying lengths for the gradient to flow back to the early layers. The identity connections form a "gradient superhighway," allowing the loss function to directly supervise even the earliest layers. This effect, sometimes called ​​implicit deep supervision​​, is a key reason why every part of a very deep ResNet can learn effectively.

The journey of the ResNet, from a simple trick to fix a training problem to a deep connection with continuous dynamical systems, showcases the beauty of scientific discovery in AI. It is a story of how shifting our perspective from multiplication to addition allowed us to build our skyscrapers of thought taller than ever before.

We have seen how the simple, almost naive-looking idea of a skip connection—adding the input back to the output of a layer—miraculously solved the problem of training profoundly deep neural networks. But the story of the Residual Network, or ResNet, does not end there. In fact, that is just the beginning. The true magic of this architecture is not just that it works, but why it works. As it turns out, the creators of ResNet, in solving an engineering puzzle, had stumbled upon a fundamental principle that echoes across vast and seemingly disconnected fields of science. The journey to understand these connections is a marvelous illustration of the unity of scientific thought, taking us from computer science to physics, from chemistry to biology.

What if I told you that a deep neural network is not just a static sequence of computations, but a simulation of a physical system evolving through time? This is the profound insight that the ResNet architecture reveals.

Consider a single residual block: the output $x_{k+1}$ is the input $x_k$ plus some transformation, $x_{k+1} = x_k + F(x_k)$. Now, think about how physicists model the world. They often describe it with differential equations, which tell us how a state $x$ changes over an infinitesimal amount of time $dt$. A simple way to write this is $\frac{dx}{dt} = F(x)$. If we want to simulate this on a computer, we can't use an infinitesimal time step. We have to take small, finite steps, say of size $\Delta t$. The simplest way to do this is the forward Euler method: the state at the next step is the current state plus the change over that time step. Mathematically, this is $x(t + \Delta t) \approx x(t) + \Delta t \cdot F(x(t))$.

Look closely at that equation. It has the exact same form as a residual block! A ResNet, then, can be seen as nothing more than a sequence of Euler steps for simulating a differential equation. Each layer is a time step, and the "depth" of the network is simply the total simulation time.

This is a breathtaking realization. It means that the entire, centuries-old field of numerical analysis, which deals with solving differential equations, can be brought to bear on designing and understanding neural networks. We are no longer just stacking layers; we are choosing a numerical integration scheme.

This perspective immediately clarifies why ResNets are so stable. The "+1" in the gradient calculation that we saw earlier is a feature of a stable numerical method. More than that, we can use ideas from linear algebra and numerical analysis to engineer even better blocks. For instance, we can introduce a scaling factor $\alpha$ to the residual branch, $y = x + \alpha F(x)$, and choose $\alpha$ to make the transformation as "well-behaved" as possible—specifically, to make its Jacobian matrix have a condition number close to 1. This is like ensuring our simulation step doesn't excessively stretch or shrink the state space, which is crucial for stable learning across many steps (layers).

And what if we use a more sophisticated numerical method? The forward Euler method is simple but has limitations. More advanced "implicit" methods, like the backward Euler method, are defined by an equation like $x_{k+1} = x_k + \Delta t \cdot F(x_{k+1})$. Notice the $x_{k+1}$ on both sides! To find the output, the layer must solve an equation for itself. While computationally heavier, these methods are immensely stable and can handle "stiff" dynamics where things change at vastly different rates. This has inspired a new class of "Deep Equilibrium Models" or "implicit layers," which push the boundaries of what is possible in network design, allowing for, in principle, a network of infinite depth with a fixed memory cost.

The connection between ResNets and differential equations is not a one-way street. If a ResNet is a numerical solver, does that mean a numerical solver is a ResNet? The answer is a resounding yes.

Consider how we simulate the diffusion of heat in a material. A common approach is to use a Partial Differential Equation (PDE) like the Heat Equation, $u_t = \alpha u_{xx}$. A standard numerical method to solve this involves calculating the state of the system, $u$, at the next time step, $u^{n+1}$, based on its current state, $u^n$. The update rule is often of the form $u^{n+1} = u^n + \Delta t \cdot (\text{change based on physics})$. This is, once again, a residual block! The "skip connection" is the physical principle of conservation—the future state is the present state plus some change.

This beautiful symmetry means we have a shared language. Scientists can use insights from deep learning to analyze and improve their simulations. Conversely, we can build new network architectures that have the laws of physics baked directly into them, making them far more efficient and accurate for scientific tasks.

A spectacular example comes from the world of quantum chemistry. The evolution of an electron's wave function, $\psi$, is governed by the time-dependent Schrödinger equation. Propagating this wave function forward in time is a central task in simulating molecular dynamics. A simple propagation step, once again, takes the form $\psi(t+\Delta t) \approx \psi(t) + (\text{update term})$. This structure is a perfect match for a ResNet layer, opening the door to using deep learning architectures to accelerate and even discover new insights in the quantum realm.

The residual principle is not just a feature of our mathematical models; it is a pattern that nature itself has discovered. The parallels are striking and offer deep, intuitive understanding.

Perhaps the most elegant analogy comes from the field of computational biology, in the folding of proteins. A protein is a long chain of amino acids (the primary sequence) that must fold into a precise three-dimensional shape to function. A ResNet is a deep stack of layers. A challenge in both is maintaining stability and integrity over a long "distance"—the length of the protein chain, or the depth of the network. Proteins solve this with mechanisms like disulfide bonds, which are strong covalent links between two amino acids that may be far apart in the sequence. This long-range "skip connection" forces parts of the chain together, drastically reducing the number of ways the protein can misfold and lending immense stability to its final, correct structure. This is precisely analogous to a ResNet's skip connection, which creates a highway for information and gradients across many layers, bypassing intermediate transformations and lending immense stability to the training of the entire network. Both are non-local links that preserve essential structure.

We can see another echo in computer vision. When a standard convolutional network processes an image, each layer of convolution tends to spread and mix information. After many layers, the crisp features from the original input can become diffuse. A convolutional ResNet combats this. The skip connection pipes a clean copy of the features from an earlier layer directly to a later one. The effect on the network's "gaze," its effective receptive field, is remarkable. The skip connection ensures that the network maintains a sharp focus on the central, most important part of its receptive field, preventing the signal from being washed out. It's a mechanism for preserving focus and identity amidst a sea of transformations.

Our journey began with a simple architectural tweak: $y = F(x) + x$. It has taken us on a grand tour of numerical analysis, scientific simulation, quantum chemistry, and molecular biology. We've learned that this is not just a "trick" for training networks. It is a fundamental principle for describing change while preserving identity. It is the language of dynamical systems, of physical conservation laws, and of biological stability.

The story of ResNet is a powerful reminder that the most profound discoveries in science often come not from inventing something entirely new, but from recognizing a deep, unifying pattern that was there all along, waiting to be seen.