Attention Sinks

Look at the attention weights of a trained transformer on almost any input, and one pattern shows up over and over: the very first few tokens of the sequence receive a large share of attention from nearly every later position, regardless of what those first tokens actually are. This holds even when the first tokens are semantically empty, a beginning-of-sequence marker, or unrelated to anything downstream. Xiao et al., 2023 named this phenomenon attention sinks and turned an odd observation into a practical fix for a real deployment problem: language models that fall apart when you try to stream-generate past their training context length with a naive sliding cache.

Why softmax needs somewhere to put the leftover weight

Softmax attention weights must sum to exactly 1 at every position (see softmax-logits); there is no option to assign low total attention when nothing is particularly relevant. If a query position genuinely has no strong match among the available keys, the softmax still has to distribute its full probability mass somewhere. Xiao et al.'s explanation is that the model learns to treat certain positions, empirically almost always the earliest ones, as a low-cost dumping ground for this excess attention mass: attending heavily to a token whose value vector contributes little useful content is a way to keep the output roughly unchanged while still satisfying the sum-to-one constraint. Early tokens make convenient sinks because, being visible to every later position under causal masking, they are the one thing every query can always reliably attend back to.

Why naive streaming breaks without them

A natural way to let an LLM generate indefinitely with a fixed memory budget is a sliding KV cache: keep only the most recent N tokens' keys and values, evict older ones as new tokens arrive (see kv-cache and sliding-window-attention). Xiao et al. found this collapses model quality catastrophically once the earliest tokens, the attention sinks, get evicted from the window. It is not that the content of those early tokens is missed; it is that the model has nowhere left to route the excess softmax mass, and the whole attention distribution destabilises across every later layer.

StreamingLLM: keep the sink, slide the rest

The fix in Xiao et al., 2023 (StreamingLLM) is almost embarrassingly simple once the mechanism is understood: permanently retain the keys and values of the first few tokens (four is often enough) in the cache, no matter how long generation runs, and slide the eviction window only over everything after them.

cache = [sink_tokens (fixed, first ~4)] + [sliding_window (most recent N)]

With the sink tokens preserved, models that were only ever trained on sequences up to length L generate stably for millions of tokens past L, at constant memory, with no retraining and no fine-tuning required. This is a striking result: a phenomenon that looks like a training artifact turns out to be load-bearing infrastructure the model depends on, and once you know to protect it explicitly, a purely inference-time cache policy recovers stability that a naive sliding window destroys.