Your Long-Context Window Still Loses the Middle
Frontier models will swallow your whole knowledge base.
Skip treating a long context window as a substitute for retrieval. Research on how language models use long contexts shows a U-shaped recall curve: facts at the start or end of a prompt are found reliably, facts in the middle are frequently missed. Position, not window size, decides retrieval. Keep retrieval; place critical facts at the edges.
The Weights Desk · 4 min read- Long context windows accept the tokens but do not reliably retrieve facts buried mid-prompt.
- The paper documents a U-shaped curve: strong recall at the start and end, weak in the middle.
- Retrieval still earns its place — to control position and cut distractors, not merely to fit the window.
- Put the load-bearing fact near the top or bottom, and measure recall on your own stack.
- Caveat: findings are from 2023-era models; frontier systems improved but positional bias isn't solved.
Verdict: skip the dump-it-all reflex
You have a 200K- or million-token window and a tempting shortcut: paste the whole knowledge base into the prompt and skip building retrieval. Skip it — as a reliability strategy. The window accepts the tokens; it does not reliably use them. 'Lost in the Middle: How Language Models Use Long Contexts' (s1) documents a U-shaped competence curve — models find facts parked at the very start or the very end of a long prompt and lose the ones buried in the middle. What decides your call is not whether the context fits. It is where the answer lands inside it.
The mechanism: primacy, recency, and a sagging middle
The authors probe this with controlled tasks — multi-document question answering and synthetic key-value retrieval — sliding the relevant item to different positions in the input (s1). Recall is highest at the edges and sags in the middle. The sharpest result: the paper reports that in some multi-document settings, burying the gold document in the middle can drop accuracy below the closed-book baseline of giving the model no documents at all. Two more findings puncture the 'just make it bigger' instinct. Longer inputs tended to lower performance. And models with explicitly extended context windows were not necessarily better at using what sat inside them.
What it means for stuffing versus retrieval
Retrieval keeps earning its place — not to make things fit, but to control position and cut distractors. Feed fewer, higher-relevance chunks. Rerank so the load-bearing fact sits near the top or the bottom, where recall is strongest, and never assume a fact is 'used' just because it is present. Here is the honest caveat: s1 tested a 2023 generation of models, and frontier systems have since posted better scores on synthetic needle-in-a-haystack retrieval. That is real progress, but it is untested at scale as a guarantee for messy, multi-fact reasoning, and the positional bias is a documented failure mode, not a closed one. Treat the middle of any long prompt as suspect until you have measured recall on your own stack.
- Can I skip retrieval if my model has a million-token context window?
- No, not for reliability. The window will hold the tokens, but the Lost in the Middle work (s1) shows facts placed in the middle of a long prompt are frequently missed. Retrieval still controls position and trims distractors.
- Where should I put the fact the model most needs to use?
- Near the very start or the very end of the prompt. The paper reports a U-shaped recall curve, with beginning and end positions retrieved far more reliably than the middle.
- Does a bigger context window fix the problem?
- Not on its own. The paper reports that models with extended context windows are not necessarily better at using information across their input, and longer inputs tended to lower performance.