← Blog

The Length of a Thought: Why Context Windows Became the New Battleground

May 31, 2026 · 4 min read

When the Transformer was introduced in 2017, its authors trained it on sentence pairs rarely longer than a few dozen tokens (Vaswani et al., Attention Is All You Need, arXiv:1706.03762). The architecture's defining mechanism, self-attention, compares every token to every other token. That is its genius and its curse: comprehension is global, but cost grows with the square of the sequence length.

For a long time, this quadratic wall defined what a language model could hold in mind at once. To understand why context windows became the central engineering obsession of the 2020s, it helps to look at the cost directly.

The quadratic wall

Self-attention computes an attention matrix of size n x n for a sequence of n tokens. Double the input, and you quadruple the compute and memory for that step. The table below makes the growth concrete (relative attention-matrix cells, normalised to the 512-token case):

Context length Attention cells (relative) Rough interpretation
512 1x a long paragraph
2,048 16x a short article
8,192 256x a research paper
32,768 4,096x a short book chapter
128,000 ~62,500x a small novel

The numbers are not a metaphor. They are the reason a naive Transformer cannot simply be handed a book. Each quadrupling of length was, for years, a quadrupling of the bill.

Three ways through the wall

Engineers attacked the problem from three directions, and modern systems borrow from all three.

1. Make the exact computation cheaper. FlashAttention did not change the math of attention; it changed how the math touches memory. By tiling the computation to keep data in fast on-chip SRAM and never materialising the full attention matrix in slower memory, it delivered exact attention with dramatically reduced memory traffic (Dao et al., 2022, arXiv:2205.14135). This is the rare optimisation that costs nothing in quality.

2. Approximate attention. A long line of work asked whether every token really needs to attend to every other token. Longformer combined local sliding-window attention with a few global tokens to reach linear scaling (Beltagy et al., 2020, arXiv:2004.05150). Earlier, Sparse Transformers factorised the attention pattern (Child et al., 2019, arXiv:1904.10509). The bet: most of the n squared comparisons carry little information.

3. Change how position is represented. Absolute position embeddings tie a model to the lengths it saw in training. Rotary Position Embedding (RoPE) instead encodes position as a rotation in the query and key vectors, which extrapolates more gracefully to longer sequences (Su et al., 2021, arXiv:2104.09864). RoPE, and methods that interpolate it, underpins much of the long-context capability shipped after 2023.

What a longer window actually buys

It is tempting to treat context length as a single number to maximise, but the useful question is what changes downstream:

  • Retrieval-augmented generation can place more evidence in front of the model per query, reducing the brittleness of top-k chunk selection.
  • Whole-document reasoning (contracts, codebases, books) becomes possible without lossy summarisation.
  • In-context learning has more room for examples, which can substitute for fine-tuning on some tasks.

But length is not comprehension. Studies of long-context models found a "lost in the middle" effect: information placed in the centre of a long input is recalled less reliably than information at the beginning or end (Liu et al., 2023, Lost in the Middle, arXiv:2307.03172). A larger window is a larger desk, not a better reader.

The honest trade-off

Every technique above trades something. Approximate attention trades a little accuracy for a lot of length. FlashAttention trades implementation complexity for speed. Position-interpolation methods trade some precision at extreme lengths. The art of a modern serving stack is choosing which trade to make for which workload.

The deeper lesson is that "context" was never free memory. It is a budget, paid in compute and attention, and the history of long-context modelling is the history of spending that budget more wisely.

Sources and further reading

← More posts Get weekly digests