Every major LLM vendor has played the same game for the past two years: announce a bigger context window. Google’s Gemini hit 1 million tokens. Anthropic’s Claude pushed past 200K. Various open-source models claim 128K, 256K, even 2 million. The numbers grow, the press releases flow, and developers build tools that assume the advertised number represents a usable working set.
It doesn’t.
A growing body of independent research — and a blunt blog post from developer Garrit Franke that has been circulating on Hacker News — confirms what many practitioners have suspected: effective context length is a fraction of the advertised number, and performance degrades gradually, predictably, and often catastrophically as you fill the window.
The Smart Zone and the Dumb Zone
Franke’s post, titled “Don’t trust large context windows,” crystallizes the issue with a simple mental model. An LLM’s context window splits into two zones. There is the smart zone, where the model processes information sharply, and the dumb zone, where attention drops off and the model starts forgetting what you told it five minutes ago.
The cutoff, based on Franke’s testing, sits somewhere around 100,000 tokens. Above that threshold, retrieval accuracy falls off a cliff — regardless of whether the model advertises 200K, 1M, or 2M tokens of context.
This is not a fringe observation. It is backed by rigorous academic benchmarks. The RULER benchmark (Hsieh et al., 2024), one of the most widely cited evaluations of long-context performance, found that the effective context length of Llama 3.1 70B — a model with an advertised 128K window — is only 64K tokens. Other models fare even worse relative to their claims.
A separate line of research has documented what is now known as the “lost in the middle” problem: LLMs exhibit a U-shaped performance curve across long contexts, reliably retrieving information at the beginning and end of a prompt while losing material in the middle. Follow-up work from MIT in 2025 traced this to fundamental properties of the attention mechanism, suggesting it is an architectural limitation, not something a quick fine-tune can fix.
Why This Matters for AI Agents
The gap between advertised and effective context is not an academic curiosity. It has real consequences for the increasingly popular paradigm of AI coding agents.
Modern coding agents — tools like Claude Code, GitHub Copilot, and Cursor — operate by loading project context into the LLM’s context window. A few file reads, a long debug session, a sprawling test run, and a single agent session can easily exceed 100,000 tokens. At that point, the agent is operating in the dumb zone, making decisions with incomplete awareness of what it was told earlier in the session.
Franke describes the dynamic bluntly: “A modern agent burns through tokens fast. A few file reads, a long debug session, a sprawling test run, and you’re at 100K before lunch.”
Some agent tools have responded with auto-compaction: when the session gets long, the agent summarizes its history and starts fresh. Claude Code uses this approach. But as Franke points out, auto-compaction has a subtle flaw: “The summary is itself produced by a model that’s already degraded.” The compaction runs after you’ve already spent time in the dumb zone.
The Mitigations That Actually Work
Franke’s recommended approach is simpler and more effective: don’t stay in the same session long enough to hit the dumb zone.
Instead of letting an agent churn through a single long session, he opens a new session and passes it a written specification. “That’s a much higher signal handoff than any automated summary,” he writes, “because I get to decide what matters going forward.”
This is the “breadcrumb” pattern: leave an artifact — a spec, a plan, a design document — that the next session or the next person can pick up cleanly, rather than relying on the model’s degraded attention to carry context forward.
Several open-source projects have formalized this approach. The `obra/superpowers` repository structures entire agent workflows around small, named artifacts: PRDs, plans, skills, sub-agent handoffs. The `mattpocock/skills` project takes a similar tack. Each artifact is a way to keep the working session in the smart zone by deliberately moving information out of the session and into something the next session can read.
The Vendor Incentive Problem
The persistent gap between advertised and effective context windows is not a mystery. The vendors have measurable incentives to inflate the number. A bigger context window is a clean metric for press releases and comparison charts. It signals progress. It sells API credits.
But the architecture behind attention — the `O(n^2)` scaling that makes processing long contexts computationally expensive — has not been fundamentally solved. Techniques like sparse attention, RoPE scaling, and Ring Attention paper over the problem. They let the model ingest more tokens without running out of memory, but they do not meaningfully extend the window over which the model can reliably retrieve information.
The Chroma research team, which has studied what it calls “context rot” — the progressive degradation of retrieval accuracy as context length increases — found that even models with state-of-the-art long-context performance show measurable degradation before hitting their advertised limits.
What This Means in Practice
For developers building on LLMs, the implications are concrete:
- Do not assume the advertised window is usable. Budget for an effective context of roughly half the advertised number, or less.
- Structure prompts with the lost-in-the-middle pattern in mind. Put the most critical information at the beginning or end of the context, not in the middle.
- Design agent workflows around short sessions with written handoffs, not long sessions with implicit state.
- Treat auto-compaction as a safety net, not a solution. The summary carries the same degradation as the session it summarizes.
The large context window arms race is not over. Model architectures will improve. But for now, the safest assumption is that your LLM can reliably work with about 100,000 tokens — and everything above that is a gamble.
Sources: Garrit Franke’s blog post “Don’t trust large context windows” (garrit.xyz), RULER benchmark (Hsieh et al., arXiv 2404.06654), Chroma research on context rot, the “lost in the middle” paper (Liu et al., 2023), and MIT follow-up work on attention limitations.