Context Rot: Why Bigger Context Windows Don't Mean Better Retrieval

In July 2025, a small team at the vector database company Chroma ran an experiment that should not have been interesting. They took a short sentence, buried it inside documents of increasing length, and asked eighteen frontier language models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, to find it. This is the "needle in a haystack" test, and by 2025 it was considered a solved problem; most vendors' launch blog posts showed a clean green grid of near-perfect scores stretching across the entire advertised context window. Chroma's twist was small: instead of only measuring whether models could find sentences that shared vocabulary with the question, they also measured performance on questions that required understanding what the sentence meant. On that second, only slightly harder task, accuracy fell steadily as the input grew, well before any model ran out of room (Chroma Research, 2025, Context Rot: How Increasing Input Tokens Impacts LLM Performance). The context window was not full. The model was not confused about instructions. It simply got worse at using what was in front of it, the longer that context became.

Why this matters: Every retrieval-augmented generation system, coding agent, and long-document assistant built in the last two years rests on an unstated assumption, that a model treats token 500 and token 50,000 with equal fidelity. That assumption is false, and the gap between the advertised context window and the usable context window is now large enough to be the dominant failure mode in production LLM systems.

TL;DR

• Context rot is the empirically observed decline in model output quality as input length grows, distinct from simply running out of context window space (Chroma Research, 2025).
• The degradation is not primarily about hitting a hard token limit. Models fail on tasks well inside their advertised window, sometimes at 10 to 20 percent of the maximum claimed length.
• Task difficulty matters more than length alone: near-perfect lexical needle-in-a-haystack retrieval coexists with steep degradation on semantic matching, distractor-heavy retrieval, and multi-hop reasoning over the same context sizes.
• The classic "lost in the middle" U-shaped curve, where models favor the start and end of a context, does not always reproduce; Chroma's own experiments found no consistent positional advantage, suggesting more than one mechanism is at play (Liu et al., 2023; Chroma Research, 2025).
• Distractors, pieces of text that are semantically similar but wrong, hurt more than raw length does, and models differ sharply in whether they hallucinate a distractor's answer or abstain when uncertain.
• Benchmarks built specifically to remove lexical shortcuts, such as NoLiMa and RULER, show that "effective" context length is often a fraction of the number printed in a model card (Modarressi et al., 2025; Hsieh et al., 2024).
• The practical response is not a bigger window; it is context engineering, the discipline of deciding what tokens reach the model at all, often via retrieval, summarization, or orchestrator/subagent architectures that keep any single context small and focused.

At a Glance

The core loop of context rot, and the industry's answer to it, fits in one diagram: as input grows, the model's limited attention budget gets spread across more competitors, so accuracy falls, which is why the working fix is not more tokens but better curation of which tokens arrive.

flowchart LR
  Grow["Input length grows"] --> Dilute["Attention mass dilutes"]
  Dilute --> Distract["Distractors and filler compete"]
  Distract --> Drop["Retrieval and reasoning accuracy drops"]
  Drop --> Engineer["Context engineering: retrieve, compact, delegate"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class Grow blue
  class Dilute,Distract,Drop rose
  class Engineer emerald

[IMAGE: Line chart showing accuracy (y-axis) versus input length in tokens (x-axis) for a lexical needle-in-a-haystack task versus a semantic needle-in-a-haystack task, both flat-then-declining but the semantic line dropping far earlier, annotated at the divergence point]

Before Context Rot: How We Got Here

Transformers were not built for this problem. The original 2017 architecture was trained on sentence pairs and short paragraphs, and self-attention's cost scales quadratically with sequence length, so early models simply could not afford long inputs (Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762). For years, "long context" meant 512 or 2,048 tokens, and the field's engineering effort went into positional encodings and attention approximations that made longer sequences computationally tractable at all.

By 2023, that had changed. Rotary position embeddings, FlashAttention-style memory-efficient kernels, and aggressive context-extension techniques pushed commercial models from thousands to hundreds of thousands of tokens. The marketing followed immediately: 100K, then 200K, then 1M-token windows became a headline feature, sold on the promise that you could paste in a codebase, a legal contract, or a book and simply ask questions.

The first crack in that promise came from Stanford and collaborators, in a paper that coined the "lost in the middle" phenomenon: language models given twenty documents and asked to find an answer performed best when the answer was in the first or last document, and markedly worse when it was buried in positions five through fifteen, even for models explicitly built for long context (Liu et al., 2023, Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172). This was a positional story: a U-shaped curve of recall against document position.

Around the same time, Greg Kamradt published a much simpler public test, needle in a haystack, which planted a single fact inside a long document and swept both the document length and the needle's depth, producing a heatmap that became the de facto industry scorecard for context length claims. By 2024, most flagship models scored close to 100 percent on this test across their full advertised window, and that heatmap started appearing in vendor launch posts as evidence that long context "worked." Researchers who looked closer disagreed. NVIDIA's RULER benchmark added multi-hop tracing, aggregation, and variable numbers of needles, and found that models achieving perfect NIAH scores still degraded sharply on these harder variants, with many models' effective context length falling to a fraction of what they claimed (Hsieh et al., 2024, RULER: What's the Real Context Size of Your Long-Context Language Models?, arXiv:2404.06654). LongMemEval, targeting conversational memory rather than document QA, found a roughly 30 percent accuracy drop when commercial assistants had to extract facts from realistic, sustained chat histories rather than short, focused excerpts (Wu et al., 2024, LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, arXiv:2410.10813). Adobe Research's NoLiMa went further, stripping out the lexical overlap between question and answer that made the original needle test easy, and watched GPT-4o's accuracy fall from 99.3 percent on short contexts to 69.7 percent at 32K tokens, a length well inside its advertised window (Modarressi et al., 2025, NoLiMa: Long-Context Evaluation Beyond Literal Matching, arXiv:2502.05167).

Chroma's 2025 report synthesized this into a single framing: it is not that models run out of room, it is that their processing of context is non-uniform, and that non-uniformity gets worse as length grows, across a wide variety of simple tasks.

timeline
  title Evolution of long-context evaluation
  2017 : Transformer trained on short sequences, quadratic attention limits length
  2023 : Lost in the Middle documents a U-shaped positional bias
  2023 : Needle in a Haystack becomes the default marketing benchmark
  2024 : RULER and LongMemEval show effective context trails claimed context
  2025 : NoLiMa removes lexical shortcuts, Chroma names the pattern context rot

[IMAGE: Annotated timeline graphic pairing each year above with a screenshot-style mockup of the benchmark it introduced, e.g. a NIAH heatmap next to 2023 and a RULER task-type table next to 2024]

How Context Rot Actually Works

The attention budget is fixed, not elastic

Self-attention assigns each query token a probability distribution over every key token in the context, computed as \(\operatorname{softmax}(QK^\top / \sqrt{d_k})\). That distribution always sums to one. This is the mechanical root of context rot: attention is a conserved resource, not an expandable one. Adding more tokens to the context does not give the model more attention to spend, it gives it more candidates to spend the same fixed budget on.

For a query attending over \(N\) keys, the weight on any single key \(j\) is:

\[\alpha_j = \frac{\exp(e_j)}{\sum_{k=1}^{N} \exp(e_k)}\]

As \(N\) grows, the denominator accumulates more terms. If the relevant token's raw score \(e_j\) does not grow proportionally, and there is no architectural reason it should, its share of the total attention mass shrinks purely from having more competitors, independent of whether those competitors are relevant. This is attention dilution, and it is a direct consequence of softmax normalization, not a bug in any particular model's training.

Distractors do more damage than length alone

Dilution alone would predict smooth, gradual decay. What Chroma's researchers found was sharper: performance held up reasonably well against pure filler (irrelevant but unrelated text), and dropped much faster once the filler was replaced with distractors, text that was semantically adjacent to the correct answer but wrong. In their experiment, a "needle" (writing advice attributed to a college classmate) was tested alongside zero, one, or four distractors (similar-sounding advice attributed to a professor, or about a different writing style). Accuracy fell both as input length grew and, independently, as the distractor count rose, and the two effects compounded. This matters because real documents are full of near-duplicate structure: quarterly reports with the same boilerplate and different numbers, contracts with similar clauses across different parties, codebases with repeated patterns and small critical differences. Those are exactly the conditions distractor experiments simulate.

A second, uncomfortable finding from that experiment was in how models failed. When the correct needle was crowded out by four distractors, GPT-family models tended to hallucinate, confidently answering with a distractor's content as though it were correct. Claude models were comparatively more likely to abstain, saying they were uncertain rather than guessing. Neither behavior is free of cost in production: hallucination silently corrupts downstream results, while excessive abstention makes an assistant less useful even when the truth is retrievable.

Position matters, but not in the way the "lost in the middle" story alone predicts

Liu et al.'s original U-curve, best at the edges, worst in the middle, was replicated broadly enough by 2024 that "put important information first or last" became standard prompting advice. Chroma's team specifically tested for it and did not find a consistent version of the curve; sweeping needle position from start to end of the context showed no reliable advantage for any particular slot in their setup. A separate, genuinely counterintuitive experiment made the picture stranger still: they placed the same needle either inside a coherent, logically flowing essay (where it stood out as a non sequitur) or inside a shuffled bag of unrelated sentences (where it should, intuitively, blend in more). Models retrieved it slightly better from the shuffled, incoherent haystack. That is difficult to square with any simple "attention decays with distance" story, and is a strong hint that positional bias, distractor interference, and something like a preference for locally coherent narrative flow are separate mechanisms that combine differently depending on task and model family, rather than one universal curve.

Conversational memory rots too, and irrelevant history is the culprit

LongMemEval separated two conditions that are easy to conflate: a "focused" context containing only the roughly 100 tokens of genuinely relevant conversation history, and a "full" context padded with up to 120,000 tokens of unrelated prior conversations. Every Claude model tested performed markedly worse in the full condition. The information needed to answer correctly was present in both conditions; the only difference was how much irrelevant material surrounded it. This is the conversational-agent version of the distractor problem: a chat assistant that accumulates history indefinitely is not gaining memory, it is accumulating noise that competes for the same fixed attention budget as the fact that actually matters.

A minimal task exposes the problem without any reasoning involved at all

Perhaps the cleanest demonstration removed reasoning and retrieval from the picture entirely. Chroma's "repeated words" task asked models simply to reproduce a long string of repeated tokens verbatim, a task with no semantic content to misunderstand. Every model's accuracy still fell as the input grew, and the failure modes were idiosyncratic rather than a graceful degrading blur: some Claude models refused outright, citing copyright concerns about reproducing text verbatim, while some Gemini models emitted content unrelated to the input entirely. Even the most mechanical possible task, "repeat this back," is not immune to length effects, and the concrete failure mode differs by model family in ways that a single accuracy number cannot capture.

[IMAGE: Grouped bar chart comparing four model families on the repeated-words task at short vs. long input, with failure-mode annotations: refusal, off-topic generation, and correct repetition, shown as stacked segments]

graph TD
  Query["Query vector"] --> Scores["Similarity score per key"]
  Scores --> Softmax["Softmax normalization"]
  Softmax --> Budget["Fixed attention budget, sums to one"]
  Budget --> Relevant["Weight on the relevant token"]
  Budget --> Noise["Weight on distractor tokens"]
  Budget --> Filler["Weight on filler tokens"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class Query blue
  class Scores,Softmax purple
  class Relevant teal
  class Noise,Filler amber

Seeing It in Motion

Two more views clarify how teams work around the problem in practice, and what a context window's health looks like as a running system uses it up.

The first is architectural: instead of appending every tool call and result to one ever-growing context, an orchestrator agent delegates focused subtasks to subagents, each with its own small, clean context, and only the condensed result flows back up.

sequenceDiagram
  participant User
  participant Orch as Orchestrator
  participant Suba as Subagent
  participant Api as ToolAPI
  User->>Orch: Long-running research task
  Orch->>Suba: Delegate one focused subtask
  Suba->>Api: Call search or read a file
  Api-->>Suba: Raw result, potentially large
  Suba-->>Orch: Condensed finding only
  Orch->>Orch: Merge findings into compact state
  Orch-->>User: Final answer

The second view is a lifecycle: a single context window, watched across a long session, moves through recognizable phases from fresh to unreliable, and the fix is a deliberate transition back to a smaller state rather than letting the window fill until it breaks.

stateDiagram-v2
  [*] --> Fresh
  Fresh --> Functional: More turns added
  Functional --> Diluted: Distractors and filler accumulate
  Diluted --> Saturated: Approaching token limit
  Saturated --> Unreliable: Retrieval fails silently
  Unreliable --> Fresh: Compaction, summarization, or reset
  Saturated --> [*]

[IMAGE: Before/after screenshot comparison of an agent transcript: left panel shows a bloated context with full tool outputs appended verbatim, right panel shows the same task with an orchestrator/subagent pattern and a condensed running summary, byte counts labeled on each]

By the Numbers

The figures below come from the benchmarks and reports cited; where a source describes a range or a qualitative trend rather than a single number, that is noted.

A second table separates the two things people conflate when they say "long context doesn't work": running out of the advertised window, versus degrading well before it.

[IMAGE: Log-scale scatter plot with context length on the x-axis and accuracy on the y-axis, one line per benchmark (NoLiMa, RULER, LongMemEval), annotated with the model's advertised maximum context length as a vertical dashed reference line per model]

A Concrete Example

Consider a support-engineering assistant built over a company's incident postmortems, a realistic case that mirrors the distractor mechanism described above without repeating any single paper's exact numbers. The knowledge base holds 40 postmortems for database outages, most following a similar template: symptom, root cause, remediation, follow-up items. An engineer asks: "Why did the checkout service go down last March?"

At the retrieval layer, a semantic search returns the correct postmortem (the March checkout outage) along with three others that are lexically and structurally similar: a checkout outage from a different month, a database outage in an unrelated service, and a remediation doc that reuses similar wording about "connection pool exhaustion." All four go into the context, in the order retrieval ranked them, totaling around 18,000 tokens once surrounding conversation history and system instructions are added.

Walking through what the mechanism predicts: the query and the correct document share some vocabulary ("checkout," "outage") but the actual causal detail ("a misconfigured retry policy tripled load during a routine failover") does not share surface wording with a plausible paraphrase of the question. This is a semantic-matching case, not a lexical one, so the dilution effect from the distractor documents bites harder than it would on a simple keyword lookup. The three similar-but-wrong documents are exactly the kind of distractors that measurably hurt accuracy more than equivalent-length irrelevant filler would.

Two outcomes are both plausible failure modes, not hypothetical curiosities: the assistant could confidently cite the wrong postmortem's root cause (the hallucination pattern GPT-family models showed more often under distractor load), or it could hedge with "I found several related incidents but I'm not certain which one you mean" even though the correct document is sitting in its context (the abstention pattern Claude models showed more often). Neither is a training failure in the conventional sense; both are consequences of a fixed attention budget being split across four documents that resemble each other far more than they resemble random text.

The mitigation that follows directly from the mechanism, rather than from adding more tokens, is to shrink the candidate set before it reaches the model: rerank the four retrieved documents against the specific date mentioned in the query ("last March"), filter out the other three before they ever enter the context, and pass in one document instead of four. This does not require a larger window; it requires better curation of what enters the existing one, which is the practical thesis of context engineering.

Where It Breaks

Context rot is not a single clean failure mode, and treating it as one leads to mitigations that target the wrong mechanism. A few specific traps:

Assuming NIAH scores predict production reliability. A model that scores 100 percent on lexical needle-in-a-haystack can still fail badly on semantic retrieval, multi-hop reasoning, or distractor-heavy documents at the same input length. Vendor-reported NIAH heatmaps measure the easiest version of the problem.

Assuming the U-shaped positional curve is universal. It was real in the original study and in several follow-ups, but Chroma's team specifically tested for it and did not find a consistent version in their setup, and the shuffled-haystack result (better retrieval from incoherent text) actively cuts against a pure distance-decay explanation. Treat "put it first or last" as a heuristic that sometimes helps, not a guaranteed fix.

Assuming more retrieved context is strictly safer than less. A common RAG instinct, when unsure how much context is enough, is to retrieve generously and let the model sort it out. The distractor findings suggest the opposite: extra retrieved passages that are topically adjacent but wrong actively compete with the correct passage for attention, so retrieving ten documents when three would answer the question can measurably hurt accuracy rather than provide a safety margin.

Treating conversational memory as free. Appending full chat history indefinitely, on the assumption that "the model can just ignore what's irrelevant," runs directly into the LongMemEval finding that irrelevant history measurably degrades extraction of the relevant fact, not just wastes tokens.

Conflating hard truncation with soft degradation when debugging. If an agent's accuracy drops on a long task, the first instinct is often to check whether it hit the token limit. Often it did not; it degraded well inside the limit, and the fix is pruning or restructuring the context rather than requesting a model with a bigger window.

Alternative Designs

There is no single fix for context rot, because it has more than one cause. The main architectural and workflow responses trade off differently:

None of these is a strict replacement for the others; production systems typically combine RAG or reranking for knowledge access with summarization or subagent delegation for session-level state.

How It Is Used in Practice

Anthropic's engineering team formalized much of this response under the label context engineering, describing it as curating and maintaining the optimal set of tokens available to a model at inference time, rather than simply writing a good prompt once. Their own guidance for long-running coding and research agents recommends the orchestrator/subagent pattern shown above precisely because it prevents the naive alternative, appending every tool call and its full output to a single growing transcript, from silently degrading the agent partway through a long task.

Production RAG systems built on vector databases like Chroma, Pinecone, or Weaviate increasingly treat "retrieve fewer, better passages" as a tuning target distinct from "retrieve everything plausibly relevant," precisely because the distractor mechanism means over-retrieval has a real accuracy cost, not merely a token-budget cost. Coding agent frameworks that operate over large repositories (tools built around Claude Code and similar agent harnesses) lean on file-level retrieval and targeted greps rather than loading entire codebases into context, for the same reason: a 200,000-line repository pasted wholesale is not "more information," it is thousands of near-duplicate distractors (boilerplate imports, repeated patterns, similar function signatures) competing for the same fixed attention budget as the three files that actually matter to the current task.

Enterprises building long-document assistants over regulatory filings, contracts, or clinical notes, domains explicitly called out by both the RULER and Chroma findings as containing highly templated, distractor-rich text, have converged on hybrid pipelines: coarse retrieval to narrow a large corpus, a reranking or filtering pass to cut down false-positive near-duplicates, and only then a generation step with a deliberately modest context rather than the largest window available.

Insights Worth Remembering

• Context rot is a resource-competition phenomenon, not a capacity-exhaustion phenomenon; softmax's fixed budget, not the token counter, is the mechanism worth reasoning from.
• A model can be simultaneously excellent at long-context lexical retrieval and poor at long-context semantic retrieval; a single NIAH score does not describe "long-context ability" as one number.
• Distractors are more dangerous than an equivalent amount of pure noise, because they compete for the same attention mass the correct answer needs, rather than simply diluting it.
• The choice between hallucinating a wrong answer and abstaining under uncertainty is itself model-family-specific behavior, and worth testing for directly rather than assuming.
• The classic U-shaped "lost in the middle" curve is real in some setups and absent in others; treat positional heuristics as conditionally useful, not as a law of transformer attention.
• Retrieving more documents "to be safe" can reduce accuracy rather than protect it, once the extras are similar enough to the correct answer to act as distractors.
• The most mechanical possible task, verbatim repetition, still degrades with length, which rules out "the model is confused about the question" as a complete explanation and points back to the attention mechanism itself.
• Context engineering: deciding what tokens reach the model, is now a distinct discipline from prompt engineering, and increasingly the higher-leverage one for long-running agents.

Open Questions

What causes the shuffled-haystack result, where models retrieved a needle slightly better from an incoherent bag of sentences than from a coherent essay where it stood out, is not yet well explained; it is an empirical anomaly that current attention-dilution or positional-bias theories do not cleanly predict, and Chroma's team presented it as a finding, not a solved mechanism. Whether newer architectural approaches (sparse attention, hybrid state-space and attention models, and test-time memory modules) genuinely change the dilution dynamics or merely shift where the degradation curve bends is an active research question rather than a settled one; early sparse-attention work suggests a real efficiency gain but, by removing some of softmax's competitive sharpness, may introduce its own retrieval costs that have not been benchmarked as thoroughly as standard attention. It is also not yet established how much of context rot is intrinsic to the transformer architecture versus an artifact of current training distributions, which rarely include the kind of dense, adversarial, near-duplicate-document contexts that stress tests like Chroma's construct; more pretraining or fine-tuning on distractor-rich data is a plausible partial mitigation that, to date, has not been rigorously isolated as a variable in the published benchmarks. Finally, there is no agreed-upon standard metric for "effective context length" the way there is for raw context window size, and until one exists, comparing models' real long-context reliability will require reading past the marketed number to the specific benchmark and task type behind it.

Sources and Further Reading

Foundational Papers

• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762
• Liu et al., 2023, Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172

Important Follow-up Work

• Hsieh et al., 2024, RULER: What's the Real Context Size of Your Long-Context Language Models?, arXiv:2404.06654
• Wu et al., 2024, LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, arXiv:2410.10813
• Modarressi et al., 2025, NoLiMa: Long-Context Evaluation Beyond Literal Matching, arXiv:2502.05167

Technical Blogs

• Chroma Research, 2025, Context Rot: How Increasing Input Tokens Impacts LLM Performance
• Chroma Research, Context Rot GitHub repository (replication toolkit)
• Anthropic Engineering, 2025, Effective Context Engineering for AI Agents

Additional Resources

• Kamradt, G., 2023, Needle In A Haystack pressure-test repository
• Hamel Husain, 2025, annotated summary of Kelly Hong's (Chroma) Context Rot presentation