Sinusoidal Positional Encodings

The original transformer paper needed to tell attention where each token sat without adding a single learned parameter, and the answer it picked was a fixed pair of trigonometric functions per dimension: PE(pos, 2i) = sin(pos / 10000^(2i/d)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d)), added elementwise to the token embedding before the first layer. No table to train, no maximum length baked into a parameter count, just arithmetic on the position index pos and the dimension index i. It is worth understanding exactly why this particular arithmetic, because the same reasoning about frequency and relative distance reappears, sharpened, in rotary position embeddings.

Why trigonometric functions at all

Feeding the raw integer pos into the network directly would be a disaster: position 50000 dwarfs a typical embedding value, and the network would need to learn a completely different response at every scale. Sine and cosine solve this by construction, every output stays bounded in [-1, 1] regardless of how large pos gets, so the positional signal never drowns out or gets drowned out by the content embedding it is added to.

The choice of base 10000 sets a geometric progression of wavelengths across the d dimensions. Dimension pairs with small i have short wavelengths and complete many oscillations per token, so they can distinguish neighbouring positions sharply; pairs with large i have wavelengths approaching 10000 * 2*pi and change almost imperceptibly from one token to the next, so they carry coarse, long-range position. Stacking many frequencies side by side is the same trick a binary number uses, where each bit flips at a different rate; here it is continuous rather than binary, but the intent is identical, encode position at every scale simultaneously.

The linear-recombination property

The paper's specific justification for choosing sin/cos over other bounded functions is a trigonometric identity: for any fixed offset k, PE(pos + k) can be written as a linear function of PE(pos), because sin(a + b) and cos(a + b) expand into linear combinations of sin(a), cos(a), sin(b), cos(b). Concretely, there exists a fixed rotation matrix depending only on k (not on pos) that maps the encoding at any position to the encoding k steps later. The hypothesis was that attention, which is built from linear projections, could learn to exploit this structure to attend by relative offset even though the encoding is added as an absolute, per-position signal at the input. It is a real property, and it is the direct conceptual ancestor of RoPE, which later made the relative-distance guarantee exact and applied it inside the attention dot product itself rather than hoping the network rediscovers it.

Why it faded from decoder-only models

Sinusoidal encoding is added exactly once, at the input, then summed into the same vector that also carries token identity. Every downstream layer has to implicitly separate "what word is this" from "where is it" out of that single entangled vector, and re-derive relative offsets from two absolute values whenever it needs them. BERT and GPT-2 replaced it with a learned absolute position table and found empirically that a table trained end to end did at least as well within its trained length, without needing the hand-designed frequency schedule. Sinusoidal encoding still shows up in vision transformers and some audio and multimodal encoders, where its parameter-free, any-length-defined nature remains convenient.

Extrapolation looks better on paper than in practice

Because the formula is defined for any pos, you can technically evaluate it at position one million with no error. But "defined" is not the same as "useful". At positions far beyond training length, many dimension pairs land on combinations of sin/cos values the network's attention and feed-forward weights never saw during training, and the linear-recombination property only guarantees a consistent geometric relationship between encodings, not that the trained weights generalise their behaviour to unfamiliar magnitudes. In practice, perplexity degrades noticeably once you push meaningfully past the training window, foreshadowing the more general length extrapolation problem every positional scheme has to confront.

When it falls down

• Extrapolation degrades despite the closed form. The formula never errors out, but quality erodes past training length because the recombination weights were never trained on those frequency combinations.
• Position and content are entangled at the input. Because the signal is summed once and never reinjected, every layer must re-derive relative offsets from two absolute values, which is strictly more work for the network than schemes that inject relative distance directly into attention (see relative position representations).
• The base constant is a hidden hyperparameter. 10000 sets how much of the wavelength range is spent on short versus long distances; too small saturates the fast dimensions within a short context, too large wastes resolution on distances the model will rarely see.
• Injected once, not reinforced. Unlike RoPE or ALiBi, which re-apply position at every attention layer, sinusoidal encoding relies on residual connections to carry the signal deep into the network, so it is diluted rather than refreshed layer over layer.

Further reading

• Vaswani et al., 2017, Attention Is All You Need - the original formula and the linear-recombination argument for choosing sine and cosine.
• Kazemnejad et al., 2023, The Impact of Positional Encoding on Length Generalization in Transformers - benchmarks sinusoidal encoding against learned, relative, and no-encoding baselines on extrapolation.