Using a Long Context.
A model that can read 128 000 tokens may still miss the one sentence that matters if it sits in the middle.
The simplest long-context test hides a fact somewhere in a long text and asks for it: a needle in a haystack. A harder one, used by Nelson Liu and colleagues in 2023, gives a question and twenty documents of which one contains the answer, and moves that document from first to last. For the models they tested the result was U-shaped: accuracy was highest when the answer was at the very beginning or end of the context and lowest in the middle, in some settings lower than answering with no documents at all. They called it being lost in the middle.
Why the middle? One plausible contributor is what a model was trained to retrieve: in natural text, what matters for the next words is usually recent, and documents put key information first. The workshop tests that idea directly with two small models of identical architecture, trained on a synthetic lookup task — forty key–value pairs, then questions about them — that differ only in which pairs the training questions asked about.
Below: retrieval accuracy against the position of the asked-about pair (1 = first of 40), for a model whose training questions asked about every pair equally and one whose training questions mostly asked about pairs near the start or end. Both were tested with every position asked equally often.
- Describe needle-in-a-haystack and multi-document retrieval tests.
- Read accuracy as a function of the evidence’s position.
- Explain how training data can create a position bias.
- Separate a model’s context length from its effective context.
Where the evidence sits
The model trained on questions spread evenly over the forty pairs retrieves about equally well everywhere: 39% at the first position, 38% in the middle, 33% at the last — not well, for a model this small, but without favourites. The model trained on questions that mostly asked about the first and last few pairs learned exactly that: 58% at the first position and 50% at the last, but only 1% in the middle — a deep U, from nothing but the distribution of its training questions. Both saw the middle pairs in every training sequence; one was never taught to look there.
This is a mechanism, not a diagnosis of any particular large model; the causes of the U-shape in production models are still discussed, and later models trained with long-context data aimed at it reduced it considerably. The practical lessons hold. A context length is a limit, not a guarantee: the effective context — how far back a model reliably uses — must be measured, with the evidence placed at every depth. Simple needle tests are easy to pass; tests that need several facts combined, or reasoning over the whole text, are much harder (plate XXI.8).
Terms
- Needle in a haystack
- A test that hides one fact in a long text and asks for it, at many depths and lengths.
- Lost in the middle
- Lower retrieval accuracy for evidence in the middle of a long context than at its ends.
- Effective context
- How much of its context a model reliably uses, as opposed to how much it accepts.
- Position bias
- A model’s accuracy depending on where in the context the evidence sits.
Sources
- Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, “Lost in the Middle: How Language Models Use Long Contexts”, TACL 2024 (arXiv 2023).