Positional Encodings

Modern AI models, particularly Transformers, possess the remarkable ability to see relationships between data points across vast sequences. However, their core mechanism, self-attention, has a critical weakness: it is blind to order. Without a way to distinguish "man bites dog" from "dog bites man," these models fail to grasp the fundamental structure of language, DNA, or any other ordered data. This article addresses this knowledge gap by delving into positional encodings, the ingenious solutions that provide AI with a "sense of place." Across the following sections, we will embark on a journey from foundational principles to advanced applications. You will first learn about the evolution of positional encoding techniques, from crude maps to the elegant geometry of rotational embeddings. Then, we will explore how this powerful idea transcends its origins, connecting disparate scientific fields and solving complex problems in biology, medicine, and physics.

Imagine a machine that can read every word in a book simultaneously. It can see the relationships between any two words, no matter how far apart they are. This is the superpower of the ​​self-attention​​ mechanism, the engine at the heart of modern AI models like Transformers. It works by creating a query, a key, and a value for each word. To decide how much attention the word at position i should pay to the word at position j, it simply compares their "query" and "key" vectors, typically using a mathematical operation called a dot product. If the query and key are aligned, the attention is high.

But this incredible power comes with a peculiar and profound weakness: the machine is an amnesiac when it comes to order. By its very design, self-attention is ​​permutation-invariant​​. If you were to shuffle all the words in a sentence, the pairwise attention scores between any two words would remain exactly the same. The model would see no difference between "man bites dog" and "dog bites man," a catastrophic failure for any system that hopes to understand language, music, or the code of life written in DNA.

To overcome this, we must give the machine a "sense of place." We need to stamp each word with information about its position in the sequence. This is the role of ​​positional encodings​​. The journey to find the right way to do this is a wonderful story of moving from brute-force methods to solutions of remarkable mathematical elegance.

The most straightforward idea is to simply create a unique vector for each position. We could have a lookup table where position 1 maps to vector $p_1$, position 2 to $p_2$, and so on, up to some maximum length. We then add this positional vector to the word's own content embedding. These are known as ​​learned absolute positional embeddings​​, as they were used in the original BERT model. The model learns the "meaning" of each position during training.

This works, but only up to a point. This approach has two fundamental flaws.

The first is the problem of extrapolation. What happens if we train our model on sentences with a maximum length of 500 words and then, at test time, we give it a 501-word sentence? The model has never seen an embedding for position 501; it has fallen off the edge of its known world. This inability to generalize to longer sequences is a major practical limitation of learned absolute embeddings. You can design experiments to show this weakness: a model with learned embeddings may perform well on documents with a familiar number of sections but fail when asked to process documents with more sections than it was trained on.

The second flaw is more subtle: it's the tyranny of the absolute. The model learns about specific, absolute positions. It might learn that a verb at position 5 often relates to a noun at position 2. But what we truly care about in language and other sequences are relative relationships. A musical motif is defined by the intervals between notes, not its absolute starting point. A critical binding motif in a biological sequence is the same functional unit whether it starts at position 50 or position 150. An encoding scheme tied to absolute indices forces the model to re-learn these patterns for every possible location, a highly inefficient way to acquire knowledge.

To solve these problems, we need to think from first principles, like a physicist designing an instrument. We need an encoding system that is infinitely extensible and inherently relational. What kind of mathematical object allows us to compare two points, $i$ and $j$, in a way that depends only on their difference, $i-j$?

The answer lies in waves and oscillations. Let's use sines and cosines, the fundamental language of periodic phenomena. We can define the positional encoding for any position, $pos$, as a vector containing a spectrum of frequencies:

Here, $k$ indexes the different dimensions of the vector, and each $\lambda_k$ corresponds to a different wavelength. By choosing these wavelengths to form a geometric progression—from very short to very long—we give the model the equivalent of both a microscope and a telescope to examine positional relationships at multiple scales.

The true magic of this construction is revealed when we consider the dot product, the core of the attention mechanism. What is the dot product of the positional vectors for two positions, $i$ and $j$? Thanks to the beautiful trigonometric identity $\cos(A-B) = \cos(A)\cos(B) + \sin(A)\sin(B)$, the inner product $\langle PE_i, PE_j \rangle$ simplifies into a function that depends only on the relative distance, $i-j$.

This is a stunning result. By simply adding these fixed, cleverly constructed vectors to our word embeddings, we've given the model a way to "see" relative positions through the fundamental operation of attention. This ​​sinusoidal positional encoding​​ scheme solves both of our earlier problems. First, since sine and cosine are defined for any number, we can generate an encoding for any position, no matter how large, solving the extrapolation problem. Second, it provides a built-in mechanism for understanding relative positions.

The sinusoidal method is a brilliant hack. It injects absolute positional information in such a way that relative information can be easily recovered. But can we do better? Can we build the concept of "relativity" directly into the architecture of attention itself? This question has led to even more powerful and elegant solutions.

One approach, used in the Transformer-XL model, is to directly modify the attention score with learned biases that are a function of the relative distance, $i-j$. This gives the model explicit parameters to capture how content at one position should attend to content at another based purely on their offset. This makes the model robust to shifts in position and is a powerful way to handle very long sequences.

An even more beautiful idea is found in ​​Rotary Positional Embeddings (RoPE)​​. It is an idea of pure geometry. Instead of adding a positional vector, we rotate it.

Imagine the query vector for position $i$ and the key vector for position $j$. RoPE rotates the query vector by an angle proportional to its position, $\theta_i = i \cdot \omega$, and rotates the key vector by an angle proportional to its position, $\theta_j = j \cdot \omega$. Now, what happens when we compute their dot product for the attention score? Because rotation is an orthogonal transformation, a wonderful geometric property emerges: the dot product of the two rotated vectors depends only on the difference in their rotation angles, which is simply $(j-i)\omega$.

The attention score becomes inherently dependent on the relative position $j-i$. Position is no longer an additive afterthought; it is woven into the very fabric of the interaction between query and key. This is a profound shift in perspective.

This rotational structure is also naturally suited for modeling periodic signals. For data like DNA, which has a characteristic helical pitch of about 10.5 base pairs, RoPE can be configured with frequencies that align with this natural periodicity, giving the model a powerful, built-in bias to discover these patterns.

This property of having a "phase" that evolves linearly with position is what gives both RoPE and sinusoidal encodings their remarkable ability to extrapolate to sequences that are thousands of tokens long, far beyond anything seen during training. The perfect symmetry in RoPE's design—rotating both query and key according to the same rule—is crucial. Even a small mismatch in how the position-dependent phase is applied to the query and key can cause the attention signal to destructively interfere with itself and vanish over long distances.

The evolution from simple lookup tables to the geometric dance of rotary embeddings illustrates a beautiful principle in science and engineering: as our understanding of a problem deepens, our solutions often become not more complex, but more elegant, powerful, and unified.

Having explored the fundamental principles of positional encoding, we might be tempted to think of it as a clever but narrow trick, a patch applied to a specific class of machine learning models. But to do so would be like looking at the law of gravitation and seeing only a way to keep apples from floating away. The true beauty of a powerful scientific idea lies not in its initial application, but in its universality—its ability to connect seemingly disparate fields and solve problems you never thought were related. Positional encoding is just such an idea. It is the art of teaching a machine about the concept of "place," and as we are about to see, the nature of "place" is one of the most wonderfully varied concepts in science.

Our journey begins where life itself begins: with the code of life, DNA.

A sequence of DNA is the quintessential ordered list. The string ACGT means something entirely different from TCGA. For decades, bioinformaticians have used models like Convolutional Neural Networks (CNNs) that are brilliant at finding local patterns—motifs—regardless of where they appear. But what if we need to understand the global architecture of a sequence, the long-distance relationships between a gene here and a regulatory element far away? This is where Transformer architectures, powered by positional encodings, have opened new frontiers.

Imagine we are trying to predict the outcome of a CRISPR gene-editing experiment. The model needs to look at the DNA sequence around the target cut site. A Transformer, with its self-attention mechanism, can theoretically connect any nucleotide to any other. But without positional encoding, it is "permutation-equivariant"—it sees the sequence as just a "bag of nucleotides." The order is lost. By adding a positional encoding, we are essentially giving each nucleotide a unique address, restoring the sequence's essential structure and allowing the model to learn the complex, position-dependent rules that govern gene editing.

The story gets even more fascinating when we move from the relatively static genome to the dynamic world of the immune system. Your body contains a vast army of T-cells, each with a unique T-cell receptor (TCR) that can recognize a specific foreign particle, or epitope. The part of the TCR that does the recognizing, the CDR3 loop, is a protein sequence formed by a "cut and paste" genetic process that often involves random insertions and deletions (indels) of amino acids in its center.

Now, suppose we want to build a model to predict which TCR binds to which epitope—a central problem in immunology. If we use a simple positional encoding that just counts from the start of the sequence ($1, 2, 3, \ldots$), the indels will throw everything off. An insertion of just two amino acids in the middle shifts the "address" of every subsequent amino acid. The model would be hopelessly confused, unable to recognize the conserved, structurally critical parts of the sequence.

The solution is a beautiful example of "biologically-informed" engineering. Instead of a simple ruler, we create a new coordinate system anchored to stable biological landmarks. We know that the beginning of the CDR3 sequence (the "V segment") and the end (the "J segment") contain conserved motifs that are not affected by the indels. We can define a residue's position not by its absolute index, but by its relative distance from these two anchors. A residue in the V segment gets a V-anchor coordinate; one in the J segment gets a J-anchor coordinate. Now, when indels occur in the fluid central region, the coordinates of the stable, anchored residues remain unchanged! We have designed a "ruler" that is robust to the specific type of variation present in the data, allowing our model to learn the true structural basis of immune recognition.

Let's now turn our eyes from the 1D world of sequences to the 2D and 3D worlds of medical imaging. Vision Transformers (ViTs) process images by breaking them down into a sequence of patches. But how do we tell the model where each patch came from? If we unroll a 2D grid of patches into a 1D sequence, the result depends entirely on whether we went row-by-row or column-by-column. A patch that is "neighborly" in 2D space could end up far apart in the 1D sequence.

The obvious solution is to give each patch a true 2D coordinate, $(x, y)$. This could be an absolute coordinate or a relative one, where the model learns how attention should be modulated based on the relative displacement between two patches (e.g., "pay more attention to the patch immediately to the right"). This preserves the image's fundamental 2D grid structure, regardless of how we linearize it for the model.

But in science, and especially in medicine, an image is more than a grid of pixels; it is a representation of physical reality. A CT scanner or an MRI machine does not produce perfect, isotropic cubes. Due to clinical needs and scanning time constraints, the resolution along one axis is often different from the others. A single step in the z-direction (slice-to-slice) might correspond to $5$ millimeters, while a step in the x-direction is only $0.5$ millimeters. This is called anisotropy.

An encoding based on simple pixel indices is blind to this physical reality. To such a system, a step of one "unit" is the same in every direction. To build a truly robust model, we must teach it physics. The positional encoding should not be a function of the pixel index $(i, j, k)$, but of the physical coordinates $(x, y, z)$ in millimeters. This information is often stored right in the medical image's metadata (e.g., the DICOM header). By creating an encoding based on physical units, our model can understand that a tumor is, say, $10$ mm wide, regardless of whether it's represented by 20 pixels on one scanner or 100 on another. This principled approach is crucial for developing reliable medical AI that can be transferred across different hospitals and machines.

The world is not always sampled on a neat, regular grid. Many scientific datasets are messy, irregular, and beautiful in their complexity. Consider hyperspectral imaging, a technique used in remote sensing to analyze the Earth's surface. For each pixel, we get a spectrum—a measurement of reflected light at hundreds of different wavelengths. This is a 1D sequence, but the "positions" are wavelengths, and they are often irregularly spaced, with gaps caused by atmospheric absorption. Using a simple index-based positional encoding would be physically meaningless. It would treat the spectral distance between $500\text{ nm}$ and $510\text{ nm}$ as identical to the distance between $800\text{ nm}$ and $900\text{ nm}$. The correct approach is to use the wavelength value $\lambda$ itself as the basis for the position, for instance, by feeding $\sin(\omega \lambda)$ and $\cos(\omega \lambda)$ to the model.

This principle finds an even more profound application in modeling human health through Electronic Health Records (EHR). A patient's medical history is a time series of events—diagnoses, lab tests, prescriptions. But this is no ordinary time series. The time gaps between events can be enormous and irregular, from seconds to decades. Furthermore, multiple events can be recorded at the exact same timestamp. A standard positional encoding based on the raw time value would be brittle, struggling to handle the vast range of scales and unable to distinguish simultaneous events.

Here, ingenuity in positional encoding design shines. To handle heterogeneous time scales, we can use a logarithmic time warp, $\ln(1+t)$, which compresses large time gaps while preserving resolution for small ones. To handle simultaneous events, we can introduce a simple, deterministic "tie-breaker" feature—a small integer that counts the order of events within a single timestamp. By combining these ideas, we construct a positional encoding that faithfully represents the quirky, irregular rhythm of a human life as captured in medical data.

So far, we have used positional encoding to help models understand existing data. But can it help us generate new data that obeys the laws of physics or creates photorealistic worlds?

Consider Physics-Informed Neural Networks (PINNs), which are trained to find solutions to differential equations. Suppose we want to model a periodic phenomenon, like a cardiac rhythm, which is governed by a known equation. We know from Fourier analysis that any periodic signal can be represented as a sum of sines and cosines at specific harmonic frequencies. We can build this powerful prior knowledge directly into our model. Instead of a generic positional encoding, we can provide the model with a basis of sine and cosine functions whose frequencies are integer multiples of the fundamental frequency of the cardiac cycle. This gives the model a perfectly aligned "scaffolding" to construct the solution, dramatically improving learning efficiency and accuracy.

Perhaps the most mind-bending application of positional encoding lies in the field of computer graphics, with the advent of Neural Radiance Fields (NeRFs). A NeRF learns a continuous 3D representation of a scene from just a handful of 2D images. The network takes a 3D coordinate $(x, y, z)$ and a viewing direction as input and outputs the color and density at that point. The magic ingredient that allows a simple neural network to capture stunningly intricate detail—the glint of light on metal, the fuzz on a tennis ball—is positional encoding.

The positional encoding used in NeRF is a set of sine and cosine functions with exponentially increasing frequencies, $\sin(2^k \pi x), \cos(2^k \pi x)$. The number of frequency bands, $L$, determines the maximum frequency the network can represent. This has a beautiful connection to the classic Nyquist-Shannon sampling theorem. To represent high-frequency detail in a scene, you need a positional encoding with high-frequency bands. But if your input views of the scene are too sparse, you haven't sampled the scene densely enough to resolve those high frequencies. The network will then be "fooled" by aliasing, producing a blurry or artifact-ridden reconstruction. Positional encoding, in this context, is the bridge between the continuous nature of the world and the discrete samples we use to observe it.

Our journey across disciplines reveals a profound, unifying theme. Positional encoding is not a monolithic technique but a philosophy. It is the art of designing the right "ruler" for your data. Sometimes that ruler is a simple integer counter. But more often, it is a coordinate system inspired by the data's inherent structure: the stable anchors of a protein, the physical dimensions of a medical scan, the irregular wavelengths of a star's spectrum, or the harmonic frequencies of a physical law.

By carefully crafting how we describe "place," we imbue our models with a deeper, more principled understanding of the world. We enable them to see structure where others see chaos, to find stability in dynamic systems, and to create new worlds from a few sparse observations. This is the inherent beauty and unity of positional encoding—a simple idea that provides a universal language for order in a complex world.