Length Extrapolation

Length extrapolation, precisely defined, means running a model on sequences longer than anything in its training data, with no additional fine-tuning, and asking whether output quality degrades gracefully, catastrophically, or barely at all. It is a narrower question than "can this model handle long context" in general, because it isolates zero-shot generalisation from the deliberate, fine-tuned extension techniques covered in RoPE scaling: NTK-aware and YaRN. Extrapolation asks what the same weights do, unmodified, outside their training distribution; scaling methods ask how cheaply you can adapt those weights to a new target length.

How the major schemes fare with zero adaptation

• Learned absolute positions fail hard. There is no table row past the trained maximum, so there is no valid computation to run at all, not degraded output, no output.
• Sinusoidal encoding is defined at any position but degrades quickly, because the trained attention and feed-forward weights never saw the frequency combinations that appear far past training length (see sinusoidal positional encodings).
• RoPE degrades more gracefully over a moderate range but still suffers, specifically from high-frequency dimension pairs aliasing into ambiguous angles once you exceed training length.
• ALiBi extrapolates well on perplexity by design; its linear bias is well-defined and monotonic at any distance, with nothing to alias (see ALiBi).
• NoPE shows the strongest zero-shot length generalisation on synthetic algorithmic tasks in the literature that introduced it, apparently because the causal mask alone supplies enough implicit position signal (see the NoPE result).

The metric trap: flat perplexity is not the same as useful context

A model can maintain low perplexity at 16000 tokens while effectively ignoring most of that context, relying mostly on a local window and general language statistics to predict the next token. Perplexity, averaged over an entire long sequence, cannot distinguish "used the whole context well" from "used the last few hundred tokens well and got lucky that most next-token predictions don't require anything further back". This is a real and common failure mode; a scheme can win on perplexity-based extrapolation claims while still failing badly on tasks that require genuine long-range retrieval.

Needle-in-a-haystack evaluation

The standard corrective is task-based: insert one specific, checkable fact at a controlled depth inside an otherwise irrelevant long context, vary both total context length and insertion depth, and measure whether the model can retrieve the fact. This separates "the model doesn't crash or degrade numerically" (extrapolation in the stability sense) from "the model actually retrieves distant information" (extrapolation in the sense that matters for real tasks). Results across many models and positional schemes show a recurring pattern, retrieval accuracy is often worse for facts placed in the middle of a long context than for facts at the very start or very end, independent of which positional scheme is in use (see effective vs nominal context length for the broader framing).