Sliding-Window Attention

The most direct way to stop attention scaling quadratically with sequence length is to stop letting every position look at every other position. Sliding-window attention does exactly that: each position attends only to a fixed-size neighbourhood of nearby positions, however long the full sequence is. Beltagy, Peters, and Cohan, 2020 (Longformer) is the reference implementation most practitioners cite, though the idea of local attention windows predates it.

The mechanism

Instead of the full causal mask allowing position i to attend to every j <= i, a sliding window of size w restricts it further, to every j with i - w <= j <= i:

scores[i, j] = -inf   for all j > i           (causal, as usual)
scores[i, j] = -inf   for all j < i - w       (outside the window)

Each position now attends to at most w prior positions, a constant, instead of up to n prior positions. The score matrix per position shrinks from O(n) entries to O(w), and summed across the whole sequence, total attention cost drops from O(n^2) to O(n * w), linear in sequence length for a fixed window size. This is the same trick, in spirit, as causal-masking: a structural constraint on which pairs are even computed, just tighter.

The receptive field grows with depth, not width

A single sliding-window layer looks like it caps how far information can travel to exactly w positions, which sounds like a severe limitation. But transformers stack layers, and each layer's output becomes the next layer's input. Position i's representation after layer 1 already encodes information pulled in from w positions back; after layer 2, attending within a window of w around that already-mixed representation reaches, in effect, roughly 2w positions back in the original sequence; after L layers, the effective receptive field grows to roughly L * w. This is the same logic as stacking convolutional layers to grow a receptive field beyond a single kernel's width, and it is why a model with a modest window size (a few hundred to a couple thousand tokens) and enough layers can still integrate information across a genuinely long sequence, just indirectly, through composition rather than through any single layer's direct lookback.

What is deliberately given up

The tradeoff is explicit: a token far outside the window cannot be attended to directly, ever, by a single layer, no matter how relevant it might be. Information from a distant token must first get compressed into the running representations of intermediate tokens, hop by hop across layers, before a later position's window can reach it indirectly. For tasks where a single fact very early in a long document matters precisely and specifically much later, this indirect path can lose fidelity that direct global attention would have preserved. This is exactly why some architectures use a hybrid: most layers use sliding-window attention for efficiency, but a small number of layers (or a small number of designated global tokens) retain full attention, giving the model a few direct long-range paths without paying full quadratic cost everywhere. Longformer's design does precisely this, mixing local windowed attention with a handful of tokens that get full global attention.

Where it sits among the alternatives

Sliding-window attention is a strict subset of full attention: it never computes more than full attention would, only less, and its window can, in principle, be made arbitrarily wide, recovering full attention when w = n. This is different in kind from attention-sinks, which is a specific inference-time cache policy layered on top of a sliding window to preserve stability, and different from multi-query-grouped-query-attention, which reduces per-token memory cost without changing which positions are attended to at all. The three techniques address different parts of the same underlying problem, the cost of attention over long sequences, and production long-context models frequently combine sliding windows, grouped-query attention, and attention-sink preservation together rather than choosing just one.

When it falls down

• Fixed-window recall is genuinely lossy, not just indirect. The multi-layer receptive-field argument shows information can travel further than w, not that it travels losslessly. Each hop through intermediate representations is a compression step, and precise recall of a specific distant fact degrades with distance in a way full attention does not suffer.
• Window size is a real hyperparameter with real failure modes. Too small a window and even a modest number of layers cannot build an adequate effective receptive field for the task; too large a window and you have simply rebuilt most of the cost you were trying to avoid.
• It does not compose for free with training length. A model trained with a given window size has learned to route information through that specific local structure; naively widening the window at inference time (to save memory differently, say) is not guaranteed to work without retraining or fine-tuning, unlike simply extending a full-attention context.
• Global tokens reintroduce part of the cost they were meant to avoid. Hybrid designs that keep a handful of positions under full attention get back most of the long-range fidelity, but that handful still costs O(n) attention per global token, so the choice of how many, and which, global tokens is itself a real design decision, not a free upgrade.

Further reading

• Beltagy, Peters, and Cohan, 2020, Longformer: The Long-Document Transformer, arXiv:2004.05150 - local sliding-window attention with a small number of global tokens.
• Child et al., 2019, Generating Long Sequences with Sparse Transformers, arXiv:1904.10509 - an earlier, related sparse-attention pattern reducing quadratic cost.