Why Attention Needs Positions

Run "the dog bit the man" through a single self-attention layer with no positional information whatsoever, then run "the man bit the dog" through the same layer. Strip out positional encoding entirely and the two sentences, despite having opposite meanings, produce the exact same set of output vectors, just relabelled by which word each vector sits under. Attention, on its own, cannot tell these sentences apart. Every bit of order a transformer knows about, it knows because something injected that information as data.

Why, formally

Self-attention computes score(i, j) = q_i . k_j for every pair of positions, normalises those scores with softmax over j, and produces a weighted sum of value vectors v_j. None of q_i, k_j, or v_j depends on the index i or j directly, only on the token embedding that happens to sit there. Permute the input sequence by any permutation and every attention weight and output permutes identically, a property called permutation equivariance. Stacking more attention layers does not change this, and neither does adding feed-forward layers (which act identically and independently on each token position, see feedforward and SwiGLU) or layer normalisation (which also acts per token). A transformer built purely from these primitives is equivariant to input order from the first layer to the last.

Contrast with architectures where order is free

Recurrent networks process tokens one at a time in sequence; the hidden state at step t literally cannot exist without having consumed the first t tokens in order, so order is baked into the computation itself, not into any auxiliary signal. Convolutional networks over sequences use fixed-offset kernels, a kernel explicitly computes "this position combined with its neighbour two steps to the left", so order is baked into the kernel's geometry. Attention has neither property. It operates over what is structurally an unordered set of key-value pairs, weighted purely by content similarity between query and key. Anything downstream needs about order has to arrive as explicit data.

What "injected as data" looks like in practice

Two families of solution exist. One adds a position-dependent vector to the token embedding before the first layer, as in sinusoidal encodings or a learned absolute position table. The other makes the attention score itself a direct function of i and j, not just of the content vectors sitting there, as in RoPE, ALiBi, and relative position representations. Either way, order stops being a free structural property of the computation and becomes an explicit design decision, with real downstream consequences for how far training generalises (see length extrapolation).

A useful mental model

Without any positional signal, a transformer processes text roughly the way you would process a bag of words annotated with co-occurrence statistics. It can still learn an enormous amount this way, which words tend to co-occur, which topics cluster together, but it fundamentally cannot represent grammar, negation scope, or "who did what to whom", because those meanings depend on order, not merely on which words are present.

When it falls down

• Causal masking is a partial exception. In decoder-only models, the mask alone reintroduces a weak, implicit position signal even with zero explicit encoding (see the NoPE result), so the strict claim "attention needs positions" holds fully only for unmasked, bidirectional attention.
• Having positional information available is not the same as using it well. A model can have a fully intact positional signal and still fail to exploit distant context effectively (see effective vs nominal context length).
• The equivariance argument is about the operation, not a trained model's behaviour. A trained model can partially compensate for a corrupted or removed positional signal using content-based heuristics learned from its training distribution, which is why ablating position after training rarely fails as catastrophically as removing it before training would predict.
• "Needs positions" often really means "needs relative positions". Absolute position turned out to be the wrong quantity for attention to want; nearly every scheme that displaced the original absolute designs (RoPE, ALiBi, relative position representations) succeeded specifically by exposing relative distance instead.

Further reading

• Vaswani et al., 2017, Attention Is All You Need - defines the self-attention operation this concept's permutation-equivariance argument is built on.
• Kazemnejad et al., 2023, The Impact of Positional Encoding on Length Generalization in Transformers - the causal-mask-as-implicit-signal argument referenced in the caveats above.