Vol. IV · Attention Plate 4 of 9causal mask, autoregressive, teacher forcing, parallel training, lower-triangular attentionAL

The Causal Mask.

Forbid each position from looking ahead, and a sequence of T tokens becomes T honest prediction problems solved in one pass.

A language model predicts each token from the ones before it. If attention were allowed to look at every position, the model at position t could simply read token t + 1 and copy it: training loss would collapse to zero and the model would learn nothing usable for generation, where the future does not exist yet. The causal mask forbids it: before the softmax, every score from a query to a later key is set to minus infinity, so its weight becomes exactly zero. The attention matrix becomes lower-triangular.

The mask also makes training efficient. Because each position sees only its past, a single forward pass over a sequence of T tokens produces T next-token predictions, each made with exactly the information it would have during generation, and all T losses can be averaged at once. This is called teacher forcing: the model is always given the true previous tokens, never its own guesses.

Below: one attention head of micro reading a sentence: each row is a query position, each column a key. Above the diagonal, the mask has set every weight to zero. Switch to the mask itself to see which pairs are allowed.

What you will learn

Explain what the causal mask prevents.
Recognise the lower-triangular attention matrix.
Count the predictions one forward pass makes.
Describe teacher forcing.

Workshop

Attention weights of micro’s layer 2, head 4, for the sentence: rows are the reading position, columns the position read from. Everything above the diagonal is zero. The second view shows the mask itself.

Honest predictions, all at once

The sentence is 14 tokens. With the mask, each position may attend to itself and everything before it — 105 of the 14 × 14 possible pairs — and 46% of the matrix is forced to zero. This head of micro uses its allowed region mostly along the diagonal and just below it, reading the current and previous token. Whatever it reads, it cannot read the future, so the 14 predictions the pass makes — one per position — are each an honest guess, and the training loss is their average.

During generation the same model is run one new token at a time, and the mask is no longer needed for the newest token: nothing after it exists. But the earlier positions’ keys and values are unchanged by the new token — precisely because of the mask — so they can be computed once and kept. That cache of keys and values is what makes generation fast (plate XXIII.2), and it exists only because attention is causal. Encoder models such as BERT, which see a whole text at once and are trained to fill in masked words, drop the causal mask; they are not built to generate (plate V.4).

The name “teacher forcing” comes from recurrent-network training in 1989: the model is forced to continue from the teacher’s tokens rather than its own, which is fast and stable but means it never practises recovering from its own mistakes — a gap that post-training and reinforcement learning later address.

Terms

Causal mask: Setting attention scores from a position to later positions to −∞, so their weights are zero.
Teacher forcing: Training on the true previous tokens at every position, all in one pass.
Lower-triangular: A matrix whose entries above the diagonal are zero.
Encoder: A model that sees the whole text bidirectionally, without a causal mask.

The causal mask makes attention look only backwards, so one pass over T tokens trains T honest next-token predictions.