Vol. XXVIII · Interpretability Plate 3 of 9activation patching, causal tracing, interventions, circuits, residual stream, information flowAE

Activation Patching.

Swap one piece of the computation and see whether the answer comes back.

Probes find correlations; activation patching tests causation. Run the model on a clean prompt and on a corrupted one that differs in a single detail, so their outputs differ. Then rerun the corrupted prompt, but at one layer and one position replace the activation with the one from the clean run, and measure how much of the clean output returns. A large restoration means that activation carries the information that differs; scanning every layer and position traces where it flows. Meng and colleagues used this “causal tracing” to locate where facts are recalled in GPT models (2022).

The course model micro (four layers, d = 128; plate I.9) is opened up in the page. Clean: “red blue green red blue”, after which micro copies “ green”. Corrupted: “red blue water red blue”, which pushes it towards “ water”. The metric is the logit of “ green” minus that of “ water” at the last position; the workshop patches the residual stream entering each layer at each position.

Below: the share of the clean–corrupted logit difference restored by patching the clean residual into each layer (rows) at each position (columns).

What you will learn
Workshop
micro, five-token prompts. m = logit(“ green”) − logit(“ water”) at the last position. Row ℓ: the residual stream entering layer ℓ (row 0 is the embedding plus position) replaced at one position by the clean run’s. 1 means the clean difference is fully restored, 0 nothing.

Following the information

The clean run favours “ green” by 4.27 logits; the corrupted one by only 1.79. Patching at the word’s own position restores everything in the residual stream entering layer 0 (1.00) — trivially, since that is where the inputs differ — and still 0.80 entering layer 2. By the stream entering layer 3, the word’s position restores only 0.22 and the final position 0.81: between layers 2 and 3, attention has moved the information about which word to copy from where it was read to where it is used. This is the signature of a copying circuit, found causally rather than by correlation.

Patching is the workhorse of mechanistic interpretability: patch heads, MLP outputs or single neurons instead of whole residual vectors, and the map narrows from “somewhere in layer 3” to a specific component. Its subtleties are the choice of corruption (it defines what question is being asked), the choice of metric, and backup behaviour — models sometimes have redundant paths that compensate when one is patched. With those cared for, it is how circuits such as induction heads (next plate) and fact recall were mapped.

Patching the clean activation into the corrupted run (as here) asks what is sufficient to restore behaviour; patching corrupted into clean asks what is necessary. The two can disagree.

Terms

Activation patching
Replacing an activation with its value from another run to test its causal role.
Corruption
A change to the input that alters the behaviour under study.
Causal tracing
Patching scanned over layers and positions to locate information.
Circuit
A set of components that together implement a behaviour.
Activation patching tests what causes behaviour by transplanting activations; scanning it traces where information flows.

Sources

  1. Meng, Bau, Andonian and Belinkov, “Locating and Editing Factual Associations in GPT” (causal tracing), NeurIPS 2022.
  2. Vig et al., “Investigating Gender Bias in Language Models Using Causal Mediation Analysis”, NeurIPS 2020.
  3. Heimersheim and Nanda, “How to use and interpret activation patching”, arXiv:2404.15255, 2024.