Drills: Positions.
Five questions at a time from the whole volume, until position schemes and long-context arithmetic are familiar.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
The volume’s two threads come back constantly: how a model knows where a token is — added vectors, rotations, biases — and what it costs to give a model a long memory — positions it never trained on, a cache that grows with every token, and context that is accepted but not used.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Compare absolute, rotary and linear-bias position schemes.
- Predict how each behaves past its training length.
- Compute key–value cache sizes.
- Distinguish context length from effective context.
What to carry forward
Attention alone ignores order (plate VI.1), so positions are supplied: as added vectors, fixed or learned (plate VI.2); as rotations that make attention depend on relative distance (plate VI.3, RoPE); or as linear penalties on distance (plate VI.4, ALiBi). On the course corpus ALiBi extrapolated past its training length and RoPE did not (plate VI.5), but RoPE is cheaply stretched by slowing its rotations and fine-tuning briefly (plate VI.6).
Long contexts have costs and limits: windows bound the work but need sink tokens kept in view (plate VI.7); accepting a long context does not mean using it, and position biases can come from training (plate VI.8); and the key–value cache grows with every token until it can outweigh the model (plate VI.9). Vol. VII turns to the architectural changes since 2017 that answer several of these problems.
Terms
- RoPE
- Rotations that make attention depend on relative position.
- ALiBi
- Linear distance penalties on attention scores.
- Position interpolation
- Dividing positions to stay within the trained range.
- KV cache
- Keys and values of every earlier token, per layer.