Drills: Interpretability.
Five questions at a time from the whole volume, until every method comes with what it can and cannot show.
This plate draws rounds of five from the questions and predictions of the eight plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that looking inside a model takes both reading and intervening: lenses and probes show what is decodable, patching and steering show what is causal, and superposition explains why the units to read are directions rather than neurons.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Choose a method for a question.
- Separate decodable from causal.
- Explain superposition and SAEs.
- Test the faithfulness of explanations.
What to carry forward
The logit lens reads intermediate layers as predictions (plate XXVIII.1); probes show what is decodable, with controls (plate XXVIII.2); patching tests causation and traces information flow (plate XXVIII.3); induction heads copy by composing two layers (plate XXVIII.4).
Sparse features superpose in fewer dimensions (plate XXVIII.5), and sparse autoencoders try to recover them (plate XXVIII.6); steering adds a direction to change behaviour (plate XXVIII.7); and written reasoning is faithful only when tested to be (plate XXVIII.8). Vol. XXIX turns to models that see and hear.
Terms
- Logit lens
- Reading intermediate layers with the output head.
- Probe
- A classifier on activations.
- Patching
- Transplanting activations to test causation.
- SAE
- Sparse autoencoder.