Position matters: the lost-in-the-middle effect
Long-context tests that place one relevant sentence at different depths in a long document show a U-shaped curve. Accuracy is highest when the sentence is near the beginning or the end of the prompt and lowest when it sits in the middle. The effect was documented in the paper "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al.), and it still shows up in practice. A prompt that fits inside the window can therefore produce a worse answer than the same question with a shorter, cleaner context.
Task shape matters more than length
Single-fact retrieval is the easy case, and it is the case vendors demonstrate. Real work looks different:
- Multi-fact aggregation — sum totals, compare clauses or count items scattered through a long document. Each additional fact lowers the chance that all of them are used.
- Distractors — documents that mention the same entities but answer a different question. Near-misses are worse than unrelated text, because they look relevant to the retriever.
- Ordering conflicts — two passages that disagree. The model often prefers the earlier one, or blends them into a single wrong statement.
- Instruction drift — a rule stated at the top of a very long prompt is applied less reliably by the end of a long generation.
Advertised context versus effective context
A model that accepts one million tokens does not use one million tokens equally well. The usable region depends on the task, the number of distractors and the position of the evidence. Treat the effective context as something you measure for your own workload, and treat the advertised number as a hard limit that you should not fill on purpose.
Test degradation on your own documents
- Pick 20 questions whose answers are single sentences in your corpus.
- For each, build three prompts: evidence at the start, evidence in the middle, evidence at the end, with the same total length.
- Score exact matches or a strict rubric, not whether the answer sounds plausible.
- Repeat with a second document set at half the length. If accuracy is stable at half length and drops at full length, you have measured degradation rather than a bad question set.
- Repeat the middle case with five facts instead of one. The drop tells you how much aggregation your workload can take.
Run the same grid on any candidate model before you switch. A model with a smaller window can beat a larger one on your documents, and the grid is the cheapest way to find out.
What to do about it
- Retrieve instead of stuff. Send the three chunks that answer the question, not the forty that mention it. Retrieval quality beats context size for most production workloads.
- Put the question first and repeat the constraint last. Both edges of the prompt are read more reliably than the middle.
- Ask for citations. Requiring the model to name the passage it used makes ignored evidence visible in the output.
- Split aggregation tasks. Ask for one fact per call and combine the results in code, rather than asking for twelve facts in one pass.
- Measure, then choose. Run the grid above on the exact documents your users bring; published benchmark tables do not describe your corpus.
Treat long context as a tool for a specific job, not as a replacement for retrieval. The two work best together: retrieve a small candidate set, then give the model enough room to reason over it.