Vol. VI · Positions and Long Context Plate 4 of 10ALiBi, attention with linear biases, recency, slopes, length extrapolationA

Linear Biases.

Make far-away tokens a little harder to attend to, by an amount proportional to how far away they are.

Attention with Linear Biases, ALiBi (Press, Smith and Lewis, 2021), is the simplest position scheme of all. Before the softmax, each score from position i to an earlier position j has m · (i − j) subtracted from it, where m is a fixed slope that differs from head to head. Nothing is learned and nothing is added to the token vectors. A head with a steep slope attends almost only to its neighbours; a head with a gentle slope can look far back, but distant tokens must earn their attention with a stronger content match.

The slopes form a geometric sequence — for 4 heads, 1/4, 1/16, 1/64 and 1/256 — so heads cover many ranges at once. Because the penalty is defined for any distance, ALiBi has nothing to extrapolate: a token 500 positions back simply gets a larger penalty. The recorded runs of plate VI.5 show what that buys.

Below: left, the ALiBi bias added to attention scores for each of four heads, as a function of distance back from the query. Right, the attention weights a query would give to 40 earlier keys of equal content match, under each head’s bias.

What you will learn

Describe ALiBi as a distance penalty on attention scores.
Compute the slopes for a given number of heads.
See how slope sets a head’s attention range.
Explain why ALiBi extends to unseen lengths.

Workshop

Left: the bias −m_h · distance for each of 4 heads. Right: attention weights over 40 earlier positions when every key matches the query equally, so that the bias alone decides; one curve per head.

A penalty for distance

With four heads the slopes are 0.2500, 0.0625, 0.0156 and 0.0039. When all keys match equally, the steepest head gives a key 30 positions back 0.01% of its attention — essentially none — and the gentlest gives it 2.40%. Give that key an extra content match of 6 and the gentlest head’s weight on it rises to 90.8% while the steepest still gives it only 4.70%: steep heads are local whatever the content, gentle heads can be pulled far back by a strong match.

ALiBi was designed for one property — train short, test long — and on the course corpus it delivers it (plate VI.5). Its cost is a built-in preference for recency that a model cannot fully undo: information far back must fight an ever-growing penalty, which can hurt tasks that need precise retrieval from long ago. It was used in BLOOM and in MPT; most later open models chose RoPE with context-extension methods instead (plate VI.6).

The paper’s title states the result: “Train Short, Test Long”. A model trained on 1 024-token sequences with ALiBi matched, on 2 048-token evaluation, a sinusoidal model trained on 2 048 tokens, while training faster and using less memory.

Terms

ALiBi: Attention with linear biases: subtract slope × distance from each attention score.
Slope: The per-head constant m_h that sets how quickly attention falls off with distance.
Recency bias: A preference for attending to recent tokens.
Length extrapolation: Working well on sequences longer than any seen in training.

ALiBi penalises attention linearly with distance, at a different rate per head. It needs no position vectors and extends to unseen lengths, at the price of a built-in recency bias.

Sources

Press, Smith and Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation”, ICLR 2022 (arXiv 2021).