Effective vs Nominal Context Length

A model card advertises a 128000-token context window. Place one specific, checkable fact at 40 percent depth inside a 100000-token document and ask a question about it, and retrieval accuracy can fall well short of the accuracy on the same fact placed at the very start or the very end. The number on the box and the length the model actually uses well are different numbers, and the gap between them is often large enough to change how you should design a prompt.

Two different definitions

Nominal length is the maximum sequence length a model accepts without erroring or producing degenerate output, set by the positional scheme's training or scaling configuration, whether that is a RoPE base and scaling factor, an ALiBi design range, or the row count of a learned position table.

Effective length is the length up to which performance on some actual task, retrieval, multi-hop reasoning, instruction-following that references distant content, stays close to short-context performance. It is task-dependent by nature; a single model can have a long effective length for simple copy or lookup and a much shorter one for reasoning that has to combine several distant spans.

Four independent causes of the gap

• Positional scaling degradation. Even a well-executed RoPE scaling method (see RoPE scaling: NTK-aware and YaRN) trades some resolution for reach; the signal at extreme lengths is a blurrier version of what the model trained on, not an identical one.
• Attention dilution. Softmax normalises over every visible key, so as context grows, the attention mass available for any single relevant token is spread across thousands of competitors unless the model has learned unusually sharp, specific attention patterns.
• Training data distribution. Most pretraining and fine-tuning documents are far shorter than a model's nominal context window, so the model receives comparatively little gradient signal teaching it what to do with genuinely long, sparse-signal inputs.
• "Lost in the middle." Empirically, information near the very start or very end of a context is retrieved more reliably than information placed in the middle, a pattern observed across many models and positional schemes, thought to relate to how recency effects and early-position anchoring interact with training data structure.

How it is measured

The standard diagnostic is a needle-in-a-haystack sweep: vary total context length and the depth at which a target fact is inserted independently, and plot retrieval accuracy as a two-dimensional heatmap over both. A single-number metric like perplexity at length L is necessary but not sufficient (see length extrapolation), because a model can hold low perplexity at long lengths while still failing targeted retrieval; perplexity is an average over the whole sequence and cannot distinguish "used the context well" from "mostly ignored the middle and got lucky".