Where Knowledge Lives.
Switch off one layer’s attention or one layer’s MLP, and see which facts survive.
A pretrained model completes “Proteins are made of amino” with “acids” and “The Earth orbits the” with “Sun”. Where is that stored? Meng and colleagues (2022) traced facts in GPT-style models with causal tracing — corrupting the input, then restoring one component’s activation at a time — and found factual recall concentrated in middle-layer MLPs at the last token of the subject, which they could then edit directly (ROME). Later work found the picture messier: facts are spread across layers, and editing one can disturb others.
The workshop runs the simplest version of the experiment on micro. For each of nine facts it measures the margin between the right completion and a plausible wrong one — log P(right) − log P(wrong) — first with the model intact, then with each layer’s attention switched off, then with each layer’s MLP switched off. A control row measures how much each ablation damages the model on ordinary text, because a component whose removal breaks everything tells us little about any one fact.
Below: left, the change in each fact’s margin when one component is switched off (red: the right answer loses ground). Right, the selected fact’s margin under each ablation, and the general damage each ablation does to held-out text.
- Measure a fact as a margin between completions.
- Ablate attention and MLP layers one at a time.
- Separate specific effects from general damage.
- Relate the results to causal tracing and model editing.
Dependence is not an address
Intact, micro prefers the right completion for all nine facts, by margins from 1.73 to 15.73 nats. Removing the first MLP is by far the most destructive change: it raises the loss on ordinary text by 7.23 nats per token, against 0.47 on average for the components of layers 3 and 4, and it flips 3 of the nine facts. That shows MLP 1 is needed for nearly everything, not that it stores these facts — in small models the first MLP behaves like an extension of the embedding. Of all 72 fact–ablation pairs, 5 flip the answer.
The telling cells are those where an ablation that does little general damage still moves a fact. Removing MLP 2 raises the loss on ordinary text by only 0.57 nats, yet turns “DNA is copied by DNA polymerase” into a margin of −5.54 — a preference for “ligase”. That is the pattern Meng and colleagues found at scale: a middle MLP carrying a specific association. Most other facts barely move under any single ablation except MLP 1, which suggests their support is spread across components or duplicated. Hase and colleagues later showed that where tracing localises a fact is a poor guide to where editing it works best: an ablation map is a picture of dependence, not an address.
Terms
- Ablation
- Switching off a component — here, zeroing its output — to see what depends on it.
- Margin
- log P(right) − log P(wrong): how strongly the model prefers the right completion.
- Causal tracing
- Corrupting the input and restoring one component at a time to find where a prediction is computed.
- Model editing
- Changing a stored fact by modifying specific weights, as in ROME.
Sources
- Meng, Bau, Andonian and Belinkov, “Locating and Editing Factual Associations in GPT”, NeurIPS 2022.
- Geva, Schuster, Berant and Levy, “Transformer Feed-Forward Layers Are Key-Value Memories”, EMNLP 2021.
- Hase, Bansal, Kim and Ghandeharioun, “Does Localization Inform Editing?”, NeurIPS 2023.