Vol. IV · Attention Plate 6 of 9attention patterns, previous-token head, attention sink, induction head, in-context copyingAE

Reading Attention Patterns.

Measure each of a model’s heads against a few simple templates and many of them turn out to be doing one recognisable job.

Looking at attention maps one by one is slow and easy to over-read. A better method is to score every head against simple templates. A previous-token head puts its weight on position i − 1. A head that attends to the first token regardless of content is using it as a sink: a place to park attention when there is nothing useful to read, since the softmax must put its weight somewhere. An induction head does something cleverer: at a token it has seen before, it attends to the token that followed the earlier occurrence — the basis of copying from context.

Induction is tested with a sequence of random tokens repeated twice. In the first copy nothing can be predicted; in the second, a model with induction heads can predict each token by finding its earlier twin and reading what came next. The scores below measure exactly that, on micro’s sixteen heads.

Below: left, every head of micro scored for previous-token attention and for attention to the first token, on a sentence. Right, every head scored for induction on 40 random tokens repeated twice, with micro’s loss on the first and second copies.

What you will learn

Score heads against previous-token, sink and induction templates.
Explain why attention sinks appear.
Test for induction with repeated random tokens.
Connect induction heads to copying from context.

Workshop

Left: mean attention to the previous token (blue) and to the first token (red) for each of micro’s 16 heads, over the sentence. Right: induction score of each head on a random sequence repeated twice.

Templates, not impressions

Scored this way, micro’s heads sort themselves out. One head puts 59% of its weight on the previous token — the second layer’s fourth head of plate IV.5. 6 heads, mostly in the upper two layers, send more than 30% of their attention to the very first token whatever it is: attention sinks. Large models show the same thing, often more strongly; the first position is always visible, so it becomes the default place for weight when a head has nothing to look up, and streaming systems that drop it from the cache break (plate VI.7).

Induction is where micro is weak. On 40 random tokens repeated twice, its best head scores only 0.24, and its loss falls from 14.10 nats on the first copy to 12.42 on the second — some copying, far from the near-perfect prediction of the second copy that larger models achieve. That matches what plate I.4 found with a repeated pattern. Induction heads typically form abruptly at a particular point in training and are the mechanism behind much in-context learning; plate XXVIII.4 trains a model in which they form and watches it happen.

Induction heads were named and characterised by Catherine Olsson, Nelson Elhage and colleagues at Anthropic in 2022, who tied their sudden formation during training to a bump in the loss curve and to the onset of in-context learning.

Terms

Previous-token head: A head that attends mostly to position i − 1.
Attention sink: A position, usually the first, that receives attention when a head has nothing useful to read.
Induction head: A head that attends to the token after an earlier occurrence of the current token, enabling copying.
In-context learning: Improving predictions from patterns earlier in the same text, without changing weights.

Score heads against templates: previous-token heads, sinks and induction heads recur across models. Induction heads let a model copy what followed an earlier occurrence.