Rotary Position Embeddings (RoPE)

RoPE does not add anything to the token embedding. It rotates the query and key vectors, and only the query and key, by an angle proportional to position, at every attention layer in the network. That single design choice, worked through algebraically, is why RoPE became the default positional scheme for nearly every open decoder-only model released since 2022, from Llama and Mistral to Qwen and Gemma variants.

Setting up the rotation

Split each query or key vector of dimension d into d/2 consecutive pairs. Treat each pair as a 2D coordinate, or equivalently as a single complex number x1 + i*x2. Pair i is assigned its own fixed frequency theta_i = base^(-2i/d), with base typically 10000, the same constant that shows up in sinusoidal encoding. At position m, pair i is rotated by angle m * theta_i:

rotate(x1, x2, angle) = (x1*cos(angle) - x2*sin(angle),  x1*sin(angle) + x2*cos(angle))

Every pair in the vector gets its own frequency, so the full rotation is d/2 independent 2D rotations applied simultaneously, one per pair, each spinning at a different rate.

Why the relative property is exact, not learned

Rotation matrices compose predictably: rotating by angle m*theta and then comparing against something rotated by n*theta is algebraically identical to comparing against something rotated by (n - m)*theta from a fixed reference. Applied to the attention dot product, the query at position m and the key at position n are each rotated by their own absolute angle, but their inner product depends only on the angle difference:

<R(m)q, R(n)k> = q . R(n - m) k

The absolute angles m*theta and n*theta cancel; only (m - n)*theta survives. This is the crucial difference from sinusoidal encoding's linear-recombination property, which merely makes relative structure available for the network to exploit if it learns to. RoPE makes the attention score a function of relative distance by construction, guaranteed by the trigonometric identity, with nothing left for training to discover.

What the frequency bands are doing

Pair i = 0 has theta_0 = 1, the fastest-rotating pair, completing a full cycle in about 2*pi positions, so it distinguishes neighbouring tokens sharply but repeats (aliases) over longer spans. As i grows, theta_i shrinks toward zero, and the slowest pairs rotate through a full cycle only over tens of thousands of positions, carrying coarse, long-range distance information. This is the same fast-to-slow frequency spread as sinusoidal encoding, but here it lives inside the attention score itself rather than as an additive input feature, which is exactly what later scaling methods (see RoPE scaling: NTK-aware and YaRN) exploit by treating fast and slow bands differently.

Rotation is an isometry, it preserves vector length, so RoPE never perturbs the scale of q or k the way an additive encoding can perturb embedding magnitude. And because rotation is applied fresh to q and k at every layer rather than injected once at the input, positional information is reinforced throughout the network rather than diluted by residual connections, unlike sinusoidal or learned absolute schemes.

A deliberately narrow scope

RoPE touches only q and k, never v. It changes which keys a query aligns with, not what information is actually read once alignment is decided. This also makes RoPE cheap to combine with the KV cache: a key can be rotated once when it is written to the cache and reused as-is for every subsequent query, with no need to reprocess earlier tokens as the sequence grows.

When it falls down

• High-frequency aliasing beyond training length. Fast-rotating pairs complete many full cycles quickly; run the model past its training length and these pairs land on angles indistinguishable, modulo 2*pi, from angles seen earlier at different positions, corrupting fine local structure first (see length extrapolation).
• The base frequency is baked in at pretraining. Changing base after the fact, which several scaling methods do, is itself a distribution shift the model was never trained on, and generally needs at least some fine-tuning to fully absorb.
• Available relative structure is not the same as used relative structure. RoPE guarantees the attention score can express relative distance; it does not guarantee the model learned to exploit distant relative offsets for retrieval (see effective vs nominal context length).
• Precision matters at long range. Sin/cos tables computed carelessly in low precision introduce rotation error that compounds as sequence length grows, an easy source of subtle long-context bugs.

Further reading

• Su et al., 2021, RoFormer: Enhanced Transformer with Rotary Position Embedding - the original RoPE derivation and the relative-distance proof.
• Peng et al., 2023, YaRN: Efficient Context Window Extension of Large Language Models - builds directly on RoPE's frequency-band structure to extend context.