Vol. VI · Positions and Long Context Plate 5 of 10length generalisation, extrapolation, learned positions, RoPE, ALiBi, NoPEAE

Beyond the Training Length.

Train on 64 tokens, test on 512. More context should help — and does, for some position schemes.

A language model is trained on sequences of some fixed length, and the question is what happens when it is given longer ones. A model that has truly learned to use context should do better on longer texts, because tokens late in a long sequence have more to condition on. A model whose position scheme breaks outside the training range will do worse, often much worse.

The workshop shows four recorded runs, identical except for position encoding: learned absolute positions, RoPE, ALiBi and none at all (NoPE). Each saw only 64-token sequences in training. Each was then evaluated — without any further training — on held-out text in sequences of 32 to 512 tokens. The loss plotted is averaged over every position of those sequences.

Below: held-out loss against evaluation length for the four position schemes; the training length, 64, is marked. Learned positions cannot be evaluated past 64 — the table has no rows there.

What you will learn

Define length generalisation and how to measure it.
Compare position schemes on sequences longer than training.
Explain why loss should fall with length when context is used.
Interpret a rising loss past the training length.

Workshop

Recorded models of micro’s shape trained at 64 tokens: held-out loss averaged over all positions, for evaluation lengths 32–512. Learned positions stop at 64.

More context, or more confusion

At the training length the four are close: RoPE 4.363, learned positions 4.431, ALiBi 4.473, NoPE 4.510 nats. Past it they part. ALiBi keeps improving as sequences lengthen — 4.155 at 256 tokens and 4.174 at 512 — the behaviour of a model that can use context it never saw in training. RoPE gets worse: 4.702 at 256 and 5.141 at 512, because its slow rotation planes reach angles it was never trained on and the attention scores become unfamiliar. NoPE degrades too (4.864 at 512), more gently. Learned positions simply stop: there is no row 65.

These are 1.8-million-parameter models on 64-token training, so the exact numbers are small-scale; the ordering — ALiBi extrapolates, RoPE does not without help — matches what the papers introducing these schemes reported for much larger models. Yet most large models use RoPE. The reason is the next plate: RoPE’s failure can be repaired cheaply by rescaling the rotations and training briefly at the new length, and RoPE without the repair has no recency penalty limiting what the model can retrieve from far back.

The loss is averaged over every position of a sequence, so longer sequences have more late, well-informed positions; a flat or rising curve past the training length is already a failure to use the extra context.

Terms

Length generalisation: Performing well on sequences longer than those seen in training.
Extrapolation: Applying a model outside the range of its training data — here, of positions.
Evaluation length: The sequence length used when measuring held-out loss.
Out-of-distribution positions: Positions, or position angles, the model never saw in training.

Trained at 64 tokens, ALiBi improves with longer contexts, RoPE degrades and learned positions cannot go past 64 at all. Position schemes decide whether more context helps.