Relative Position Representations

Two years before RoPE and three before ALiBi, Shaw, Uszkoreit, and Vaswani asked a narrower question: instead of encoding absolute position at the input and hoping attention can back out relative distance, what if attention's own score and value computation were handed the relative distance i - j directly, as a small learned vector? The answer, relative position representations, is the conceptual bridge between "position is a feature you add once" and "position is a quantity attention consumes at every layer", the idea every modern scheme (RoPE, ALiBi, T5's bias) inherits in some form.

The mechanism

For each attention head, a small table of learned vectors a_k^K and a_k^V is indexed by clipped relative distance k = clip(i - j, -w, w), so distances beyond a window w all collapse onto the same boundary vector. The key-side vector is added into the score computation, and the value-side vector is added into what gets read out:

score(i, j) = (q_i . (k_j + a_{i-j}^K)) / sqrt(d_k)
output_i    = sum_j softmax(score)_{i,j} * (v_j + a_{i-j}^V)

Because the table is indexed by i - j and clipped to a fixed window, its size is 2w + 1 regardless of sequence length, unlike a learned absolute position table, whose size scales with the maximum sequence length itself. This already guarantees a relative-only dependence structurally, the same property RoPE and ALiBi later deliver by different, cheaper means.

What it cost, and why cheaper schemes replaced it

The relative embedding differs for every (i, j) pair, so the extra lookups and additions scale roughly as O(n^2 * d_k), a real wall-clock cost at long sequence lengths, on top of ordinary attention. Compare that to RoPE, which precomputes a rotation table once per position, O(n * d), or ALiBi, which adds a single scalar per pair with zero learned parameters. Much of the benefit of relative position representations, attention that depends on distance rather than absolute index, turned out to be achievable far more cheaply, which is the main reason this exact formulation did not become the default at scale even though it was directionally correct.

Its direct descendant: T5's relative bias

T5 simplified the idea into something closer to ALiBi in shape: bucket relative distances logarithmically, fine-grained near the query and coarse further away, and learn a single scalar bias per bucket per head rather than a full vector per distance. That scalar bias is added straight to the pre-softmax score, much like ALiBi's penalty, except the bias values are learned rather than fixed by a slope formula. It is a genuine midpoint between Shaw et al.'s fully learned vector approach and ALiBi's fully fixed scalar approach.

The idea that survived

What generalises from this line of work, and shows up unchanged in RoPE, ALiBi, and T5's bias, is a single structural commitment: make the attention computation a function of i - j, not of i and j separately. Everything since has been about finding cheaper and more robust ways to deliver that same property.

When it falls down

Quadratic-scale overhead in relative lookups. Real wall-clock cost per layer per head at long sequence lengths, a major reason it lost out to RoPE and ALiBi in large-scale training.
The clipping window discards exactly the resolution extrapolation needs. Very long-range distances all collapse into the same boundary bucket, losing distinction precisely where length extrapolation is hardest.
Original design shares tables across layers. It does not compose as naturally with modern attention variants like grouped-query attention (see KV cache) as RoPE's per-head rotation does.
Does not solve extrapolation on its own. A model trained with window w still saturates at the far bucket boundary for distances beyond w, a softer but structurally similar cliff to a learned absolute table running out of rows.

The mechanism

What it cost, and why cheaper schemes replaced it

Its direct descendant: T5's relative bias

The idea that survived

When it falls down

Further reading