ALiBi: Attention with Linear Biases

ALiBi injects no vector anywhere. It adds one number to each pre-softmax attention score, proportional to how far back the key is, and scaled by a slope fixed per attention head:

score(i, j) = q_i . k_j  -  slope_h * (i - j)      for j <= i

No rotation, no embedding table, no learned parameters at all. It is the cheapest positional scheme in this family by parameter count, and it was introduced under the title "Train Short, Test Long", which is a fair summary of what it delivers.

The slope schedule

For n attention heads (n a power of two), ALiBi assigns slopes as a fixed geometric sequence: slope_h = 2^(-8*h/n) for h = 1..n. This spreads slopes across a wide range within a single layer, some heads get a steep slope and effectively attend only to a narrow, nearly local window; others get a shallow slope and attend almost uniformly far back. A single ALiBi layer is, in effect, an ensemble of receptive-field widths rather than one shared notion of "how far attention reaches". The slopes are fixed by head count at pretraining time and are not learned or retuned at inference.

Why it extrapolates so cleanly

The bias is a plain linear function of i - j, defined and well-behaved at any distance, with no periodic structure to alias and no lookup table to run out of rows in. Compare this to RoPE, whose fast-rotating dimension pairs wrap around and produce ambiguous angles once you exceed training length, or a learned position table, which simply has no entry past its maximum index. Press et al. showed a model trained with 512 to 1024 token sequences kept low perplexity when evaluated at 2048 tokens and beyond, a result that held up far better than sinusoidal or learned baselines trained under matched compute. Nothing about the bias formula changes character past training length; only how far the model was ever forced to exercise that reach during training goes untested.

What is given up for that robustness

The bias is monotonic and additive: it makes every key further back strictly less attractive to every query, regardless of content. That is a strong, useful prior for language modelling, where recent context usually matters more, but it actively works against tasks that need a query to attend sharply to one specific token far in the past while ignoring everything in between, exactly the shape of a retrieval or needle-in-a-haystack task. RoPE's relative rotation, by contrast, does not penalise distance by construction; the network is free (subject to its own extrapolation limits) to attend sharply anywhere.

ALiBi shipped in BLOOM, MPT, and a handful of early long-context experiments, but lost ground to RoPE plus explicit scaling (see RoPE scaling: NTK-aware and YaRN) as long-context benchmarks shifted toward precise retrieval rather than raw perplexity stability. Later comparisons found much of ALiBi's extrapolation advantage narrows once RoPE is paired with a well-tuned scaling method, and the field mostly converged on RoPE as the base scheme, adding scaling on top rather than switching to a monotone bias.

When it falls down

• Structurally discourages sharp long-range recall. The recency penalty is a content-independent tax on distance; tasks needing exact recall of one far-back fact fight the bias rather than being helped by it.
• Slope schedule is fixed at pretraining. Because slopes are set by head count, adapting them to a very different deployment range after the fact is not a lightweight inference-time knob the way RoPE's rotation base is.
• Interacts awkwardly with windowed or attention-sink inference tricks. Streaming-inference schemes that prune old keys (see KV cache and context windows and long context) need to account for how the fixed bias magnitude changes when the visible window itself changes shape.
• Narrower advantage than originally reported once RoPE is scaled well. Head-to-head comparisons against tuned RoPE-scaling methods often close most of the perplexity-extrapolation gap that motivated ALiBi in the first place.

Further reading

• Press et al., 2021, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation - the original ALiBi paper, slope schedule, and extrapolation experiments.