The Sum-Over-States Formula: From Quantum States to Material Properties

When a force is applied to an object, it responds. A spring compresses, a ruler bends, and on a microscopic level, a molecule's electron cloud deforms in an electric field. But how can we predict the extent of this response from first principles? Why are some molecules "squishy" and easily polarized, while others are "rigid"? This fundamental question in chemistry and physics finds its answer in one of quantum mechanics' most insightful expressions: the sum-over-states formula. This formula, born from perturbation theory, provides a definitive link between the hidden, internal world of a molecule's quantum states and the tangible, measurable properties it exhibits. It reveals that a molecule's response is a symphony of virtual possibilities, a dynamic interplay between all its available excited states. In the chapters that follow, we will embark on a journey to understand this powerful concept. First, in "Principles and Mechanisms," we will explore the quantum mechanical origins of the formula, dissecting its components to understand the story it tells. Then, in "Applications and Interdisciplinary Connections," we will see how this single theoretical tool unlocks a unified understanding of a vast array of phenomena, from simple polarizability to the exotic behavior of modern materials.

Imagine you push on a spring. It compresses. You bend a plastic ruler. It flexes. In our everyday world, we have an intuitive sense that objects deform in response to a force. What is remarkable is that the atoms and molecules that make up these objects do the same thing, just on a fantastically small scale. When a molecule finds itself in an electric field—perhaps from a nearby ion or a passing light wave—its fluffy cloud of electrons gets pushed one way and its positively charged nuclei the other. This separation of charge creates a temporary, or ​​induced​​, dipole moment. The ease with which this happens is a fundamental property of the molecule called its ​​polarizability​​.

But how does a molecule "decide" how much to polarize? Is it a soft, squishy molecule or a hard, rigid one? The answer lies deep within the rules of quantum mechanics. It’s not a simple story of an electron cloud being pulled like taffy. Instead, it’s a beautiful and subtle tale of possibilities, virtual jumps, and energy costs. Our goal in this chapter is to unravel this story and arrive at one of the most insightful formulas in quantum chemistry: the ​​sum-over-states formula​​.

Solving the Schrödinger equation for a molecule sitting in an electric field is, to put it mildly, monstrously difficult. We can't do it exactly. So, we turn to one of the most powerful tools in a physicist's toolkit: ​​perturbation theory​​. The idea is simple. We first solve the problem we can solve—the isolated molecule with no field, described by a Hamiltonian $H_0$. Then, we treat the electric field as a small disturbance, a "perturbation," $H'$.

Perturbation theory tells us how the molecule's energy levels and wavefunctions change due to this disturbance. The first change we might look for is the first-order correction to the ground state energy, $E^{(1)}$. This term turns out to be proportional to the electric field strength, $\mathcal{E}$, and is related to the molecule's permanent dipole moment. But many molecules, like hydrogen ($\text{H}_2$) or methane ($\text{CH}_4$), have no permanent dipole moment due to symmetry. Does this mean they don't respond to a field? Of course not!

This is where the next level of the story unfolds: the ​​second-order energy correction​​, $E^{(2)}$. This energy shift is proportional not to $\mathcal{E}$, but to $\mathcal{E}^2$. From classical physics, we know the energy of an induced dipole in a field is given by $-\frac{1}{2} \alpha \mathcal{E}^2$. By comparing the classical definition with the quantum mechanical result from perturbation theory, we find a direct link: the polarizability, $\alpha$, is precisely what determines this second-order energy lowering. The molecule rearranges its electrons to lower its energy in the field, and the polarizability tells us by how much. This process leads us directly to the star of our show.

By working through the mathematics of second-order perturbation theory, we arrive at a magnificent expression for the static polarizability of a molecule along the z-axis, $\alpha_{zz}$:

This is the famous ​​sum-over-states formula​​. At first glance, it might seem like a daunting collection of quantum symbols. But let’s not be intimidated. This equation is telling a story, and our job is to listen. It says that the polarizability isn't one single thing, but a sum—a symphony, really—of contributions from all the excited states $|\psi_k\rangle$ of the molecule. Let's look at the parts of each term in the sum.

​​The Numerator: The 'Ticket' for a Virtual Jump​​

The term in the numerator, $|\langle\psi_0|\hat{\mu}_z|\psi_k\rangle|^2$, is the square of the ​​transition dipole moment​​. This quantity is the master key. It determines the probability of a transition between the ground state $|\psi_0\rangle$ and an excited state $|\psi_k\rangle$ when driven by light. In the context of polarizability, you can think of it as a measure of how effectively the electric field can "connect" or "mix" the ground state with that particular excited state. If this term is large, the field can easily coax the molecule into a "virtual" excursion to state $|\psi_k\rangle$. If it's zero (a "forbidden" transition), that state cannot participate in the symphony; it contributes nothing to the polarizability.

This brings us to a wonderfully deep connection: the very same excited states that are responsible for a molecule's color—that is, the ones it can absorb light to get to—are the same states that govern how much it deforms in a static electric field. A molecule with strong absorptions in its spectrum tends to be highly polarizable.

​​The Denominator: The 'Price' of the Ticket​​

The denominator, $E_k - E_0$, is simply the energy difference between the excited state and the ground state. This can be thought of as the "energy cost" for the molecule to make that virtual jump to state $|\psi_k\rangle$. Notice that it's in the denominator. This means that states with a low energy cost—those that are closer in energy to the ground state—make a larger contribution to the polarizability. This is perfectly intuitive. The molecule prefers to use "cheaper" virtual states to rearrange itself and lower its energy in the field.

So, the whole picture comes together. The polarizability of a molecule is the sum of all its virtual possibilities. The molecule, when pushed by a field, doesn't actually jump to an excited state. Instead, it "borrows" a little bit of character from all the excited states it's allowed to talk to. The amount it borrows from each state is determined by the strength of the connection (the numerator) and inversely by the energy cost (the denominator). It's a quantum dance of compromise, mixing in just the right amount of each excited state to find the most stable configuration in the presence of the field.

Our story so far has been about static, unchanging electric fields. But what happens when the field is oscillating, like a wave of light? The situation becomes even more interesting. The polarizability is no longer a constant; it becomes dependent on the frequency, $\omega$, of the light. We call this the ​​dynamic polarizability​​, $\alpha(\omega)$.

The sum-over-states formula adapts beautifully to this new scenario. For a simple two-level system, the off-diagonal polarizability $\alpha_{xy}(\omega)$ looks something like this:

Look at that denominator! Here, $\omega_{eg}$ is the natural transition frequency of the molecule. When the frequency of the incoming light, $\omega$, gets very close to $\omega_{eg}$, the denominator approaches zero, and the polarizability gets enormous. This phenomenon is called ​​resonance​​. It’s the molecular equivalent of pushing a child on a swing. If you push at just the right rhythm—the swing's natural frequency—a small push can lead to a huge amplitude. Similarly, light at a resonant frequency can induce a massive dipole response in a molecule.

The full expression for any molecule is even more revealing. It's often called the ​​Kramers-Heisenberg formula​​:

This formula, derived from rigorous linear response theory, contains a universe of physics. The term $\mathrm{i}\Gamma_n$ is not just mathematical decoration. This small imaginary part is the key to life itself, in a manner of speaking. It ensures ​​causality​​—that the molecule responds after the light wave hits it, not before. Physically, the width $\Gamma_n$ represents the finite lifetime of the excited state. And most importantly, it's what allows the molecule to actually ​​absorb​​ light. The real part of $\alpha(\omega)$ governs how the speed of light changes as it passes through a material (the refractive index), while the imaginary part governs how much light is absorbed. This single formula unifies the phenomena of refraction, dispersion (why a prism splits light), and absorption into one coherent quantum mechanical framework. The total strength of an absorption is captured by a related quantity called the ​​oscillator strength​​.

There is, however, a very large elephant in the room. The summation sign $\sum_{k \neq 0}$ instructs us to sum over all excited states of the molecule. For any real molecule, this is an infinite number of states, including the continuum of ionized states! How could we possibly perform such a sum? This is where the true art of theoretical physics comes into play.

One clever strategy is the ​​Unsöld approximation​​. The main difficulty in the sum is that each term has a different energy denominator, $E_k - E_0$. The approximation consists of a bold move: what if we replace all these different energy costs with a single, effective energy, $\Delta E_{\text{eff}}$? For a hydrogen atom, a good choice is its ionization energy. Once the denominator is a constant, it can be pulled out of the sum. The remaining summation, $\sum_{k \neq 0} |\langle k | z | 0 \rangle|^2$, can be magically simplified using a quantum mechanical identity called the ​​completeness relation​​. We can evaluate it without ever summing over a single excited state! This simple but powerful trick gives a surprisingly accurate estimate for the polarizability of hydrogen.

An even more elegant approach is the ​​Dalgarno-Lewis method​​. This technique bypasses the sum-over-states formulation entirely. It recognizes that the sum is just a formal way of writing the first-order correction to the wavefunction. So, instead of calculating the sum, why not try to solve for this corrected wavefunction directly? This leads to a differential equation which, for certain simple systems like the particle in a box, can be solved exactly. The solution gives us the polarizability as a closed-form expression, effectively performing the infinite sum for us in one fell swoop. Both of these methods beautifully illustrate that often, in physics, a change in perspective can transform an impossible problem into a tractable one.

Finally, a word of caution. The sum-over-states formula is exact only if the sum is truly performed over the complete, infinite set of states. In most modern computational chemistry, we are forced to approximate this by summing over a finite, truncated set of calculated states. This violation of ​​completeness​​ is not just a numerical error; it can break fundamental principles. For example, a calculation with an incomplete set of states can give a polarizability that unphysically depends on the choice of coordinate system. It can also fail to reproduce the correct behavior of the molecule in response to very high-frequency light.

This serves as a humble reminder. Our physical models and the formulas that emerge from them are windows into the workings of nature. The sum-over-states formula provides a profound and beautiful window into the responsive nature of quantum systems. But a true master of the craft understands not only the view through the window but also the size and shape of the frame.

In the last chapter, we delved into the heart of the "sum-over-states" formula, understanding it as the quantum mechanical recipe for how a system adjusts itself in the face of a small disturbance. It's a beautiful piece of theory, but theory for its own sake can feel like a ship in a bottle—intricate, but sealed off from the vast ocean of the real world. Now, we're going to break the bottle. We will see that this formula is not an isolated curiosity; it is a master key that unlocks a staggering variety of physical phenomena across chemistry, physics, and materials science. It is the bridge between the hidden, "internal" world of quantum states and the measurable, "external" properties that define the world around us.

Imagine a perfectly still bell. Its silent state is the "ground state." If you give it a gentle tap—a perturbation—it doesn't just produce its fundamental note. It rings with a complex sound, a superposition of its fundamental tone and a whole series of higher-pitched overtones. The character of the ring, its richness and timbre, is determined by which overtones are excited and by how much. The sum-over-states formula is physics' way of describing this ringing. It tells us that when we "tap" a molecule with an electric or magnetic field, its new state is a mixture of its original ground state and all its possible excited states, each contributing a little bit of its character. The denominator in the formula, the energy difference, is the "cost" of borrowing from each excited state—the higher the energy of an excited state, the less it contributes to the response.

The most direct and intuitive application of our formula is explaining how atoms and molecules respond to an electric field. When you place a molecule in a static electric field, its cloud of negative electrons is pulled one way and its positive nuclei the other. The molecule stretches, developing an induced dipole moment. The measure of how easily it stretches is called the ​​polarizability​​. A "squishy" molecule has a high polarizability; a "rigid" one has a low polarizability. The sum-over-states formula tells us precisely what makes a molecule squishy or rigid at the quantum level.

Let's start with the simplest possible picture: a charged particle moving on a circle. When an electric field is applied across the circle, the particle's wavefunction, originally uniform, distorts. Our formula predicts how much the ground state energy drops due to this distortion, and from this energy drop, we can extract the polarizability. We find it depends on the charge, the mass, and the radius of the circle. More importantly, the calculation shows us the mechanism: the electric field makes the ground state wavefunction mix with the excited states—specifically, those states that have the right symmetry to couple with the ground state via the dipole operator.

This is not just a feature of toy models. For a real hydrogen atom, the story is the same. The electric field causes the spherical $1s$ ground state to mix with the excited $p$ orbitals ($2p, 3p, \dots$), creating a distorted, polarized state. The sum-over-states formula involves summing up the contributions from all these excited states. In practice, we often don't need to sum to infinity. Truncating the sum to just the first few excited states, like the $n=2$ and $n=3$ states, can often give a remarkably good approximation to the true polarizability. This tells us something crucial: the response is dominated by the most accessible excited states.

When we move from atoms to molecules, the principles remain the same, but the "states" in our sum are now molecular orbitals. For a simple molecule like $\text{H}_2$ or ethene, we can use molecular orbital theory to describe the ground and excited states. The most important excitation is often the one that promotes an electron from the Highest Occupied Molecular Orbital (HOMO) to the Lowest Unoccupied Molecular Orbital (LUMO). This leads to a beautifully simple and powerful approximation:

The polarizability is proportional to the square of the transition dipole moment between the HOMO and LUMO, and inversely proportional to the HOMO-LUMO energy gap. This is a cornerstone of chemical intuition. Molecules with a small HOMO-LUMO gap (like long, conjugated organic dyes) are typically highly polarizable. They are the "soft springs" of the molecular world, easily distorted by electric fields. This simple relationship is a powerful tool for designing molecules with specific optical properties.

Furthermore, the sum-over-states formula connects polarizability directly to spectroscopy. The terms in the sum can be rewritten using ​​oscillator strengths​​, which are quantities that measure the intensity of spectroscopic transitions. This means we can estimate a molecule's polarizability by analyzing its absorption spectrum. The strongest absorptions at the lowest energies (like the HOMO-LUMO transition) will give the largest contributions to the molecule's ability to respond to a static field.

The true power of the sum-over-states framework is its universality. The perturbation doesn't have to be an electric field. Let's see what happens when we use a magnetic field.

One of the most powerful tools in a chemist's arsenal is Nuclear Magnetic Resonance (NMR) spectroscopy. The position of an NMR peak—the "chemical shift"—depends on the magnetic field experienced by a nucleus. This local field is not quite the same as the external field applied by the magnet, because the molecule's own electrons "shield" the nucleus. The sum-over-states formula explains a crucial part of this shielding, the ​​paramagnetic shielding term​​. The external magnetic field perturbs the electron cloud a second time, mixing the ground electronic state with excited states. This mixing generates a new, internal magnetic field at the nucleus that opposes or adds to the external field. The calculation reveals that the chemical shifts we measure in an NMR spectrum are a direct reporter on the energy landscape of the molecule's excited states!

The formula also explains more subtle magnetic behaviors. In molecules or materials with more than one unpaired electron (a total spin $S > 1/2$), the spin energy levels can be split even in the absence of an external magnetic field. This phenomenon is called ​​Zero-Field Splitting (ZFS)​​, and it is fundamental to the behavior of molecular magnets and the operation of certain quantum bits (qubits). Where does this splitting come from? It arises from a dance between the electron's spin and its orbital motion, a relativistic effect called spin-orbit coupling. Using second-order perturbation theory, the sum-over-states formalism shows how spin-orbit coupling provides a pathway for the ground state to mix with excited orbital states. This virtual excursion into the excited state manifold creates an effective magnetic interaction between the spins themselves back in the ground state. The formula allows us to calculate the ZFS tensor, $\mathbf{D}$, which describes this interaction, revealing its origin in the intricate interplay between spin, orbital angular momentum, and the excited state structure.

So far, we have discussed linear responses: double the field, and you double the induced dipole moment. But if the "tap" from our field is strong enough—like the intense field of a laser—the response of the material can become nonlinear. This is the realm of ​​nonlinear optics​​, which gives us remarkable technologies like frequency-doubling lasers that turn red light into green.

The sum-over-states formulism can be extended to describe these higher-order effects. The second-order susceptibility, $\chi^{(2)}$, is a tensor that captures the first nonlinear term. It links together a trio of fields and is responsible for a host of phenomena. Consider two such effects:

1. ​​The Pockels Effect:​​ A static DC electric field changes the refractive index of a crystal for a beam of light passing through it. This is described by $\chi_{ijk}^{(2)}(-\omega; \omega, 0)$.
2. ​​Optical Rectification:​​ An intense beam of light creates a static DC polarization (a voltage) across a crystal. This is described by $\chi_{kji}^{(2)}(0; \omega, -\omega)$.

These two effects seem like mirror images of each other. By writing down the sum-over-states expression for each process, we discover a breathtakingly elegant truth: the microscopic expressions for the two tensors are identical! This "Kleinman symmetry" reveals a deep and non-obvious connection, a hidden harmony in the way light and matter interact, all unmasked by the systematic perturbation expansion.

The ultimate expression of the formula's versatility is its ability to describe "cross-coupled" responses, where a stimulus of one kind produces a response of a completely different nature. A fascinating example is the ​​magnetoelectric effect​​, where applying an electric field to a material induces a magnetization, or applying a magnetic field induces an electric polarization. Such "multiferroic" materials are at the forefront of research for new types of memory and sensors.

How can such a seemingly "magical" coupling occur? The sum-over-states formula provides the microscopic recipe. For an electric field to create a magnetic moment, there must be a quantum mechanical pathway connecting the ground state back to itself via two perturbations: one by the electric dipole operator and one by the magnetic dipole operator. The sum runs over intermediate excited states that can be reached from the ground state by an electric-dipole transition, and can return to the ground state via a magnetic-dipole transition (or vice versa). The formula shows that this can only happen in materials with specific symmetries (for instance, those lacking a center of inversion), providing a clear design principle for scientists searching for these exotic materials.

From the simple stretching of an atom to the subtle shifts in an NMR spectrum, from the behavior of molecular magnets to the exotic dance of multiferroics, the sum-over-states perturbation formula provides a single, unified language. It teaches us that the observable properties of matter are not static attributes but dynamic responses, governed by a perpetual, virtual conversation between a system's ground state and the entire ladder of its excited states. Every property is a spectrum in disguise. The world we see and measure is a reflection of this unseen, restless, and beautiful quantum reality.