What Attention Weights Do Not Say.
Where a head looks is easy to see. Whether what it found mattered is a different question, answered only by taking the head away.
Attention maps are tempting explanations: the model looked at this word, so this word must be why it answered as it did. The inference is weak. A head may attend strongly to a token whose value vector is small or later cancelled; heads write into a shared stream that later layers may ignore; and a head with flat, diffuse attention can be carrying something every later layer depends on. Attention says where a head reads, not how much the reading changed the output.
The direct test is an ablation: remove a head — here, zero its output — and measure how much the model’s loss on the text gets worse. The workshop does this for each of micro’s sixteen heads and plots the damage against how sharply each head attends.
Below: for each head of micro, how sharp its attention is (horizontal) and how much worse micro predicts the sentence when that head is removed (vertical). If sharp attention meant importance, the points would rise from left to right.
- Explain why attention weights are not explanations.
- Measure a head’s importance by ablation.
- Compare importance with attention statistics.
- Recognise when an interpretation needs a causal test.
Look, then intervene
Removing the single most important head raises micro’s mean loss on the sentence from 2.979 to about 2.979 + 0.407 nats, yet that head is one of the most diffuse — its sharpness is 0.12, against 0.44 for the sharpest head. Across all sixteen heads, sharpness explains r² = 0.01 of the variation in importance: essentially none. The third layer’s first head sends 49% of its attention to the first token, a conspicuous pattern, and removing it changes the loss by −0.009 nats. 2 heads even make the prediction slightly better when removed.
The general lesson recurs throughout interpretability (Vol. XXVIII): a pattern that is easy to see is a hypothesis, and a causal test — removing, swapping or patching a component and measuring the effect — is the evidence. Ablation itself has pitfalls: zeroing a head puts the model in a state it never saw in training, and the next plates of that volume replace it with gentler interventions such as activation patching. Researchers argued over exactly this in 2019, in papers titled “Attention is not Explanation” and “Attention is not not Explanation”.
Terms
- Ablation
- Removing or zeroing a component and measuring the change in the model’s behaviour.
- Causal importance
- How much a component’s removal or change affects the output.
- Faithfulness
- Whether an explanation reflects what actually caused the model’s output.
- Sharpness
- How concentrated a head’s attention is; here one over two to the power of its entropy in bits.