Vol. XXVIII · Interpretability Plate 6 of 9sparse autoencoders, dictionary learning, features, monosemanticity, L1 penalty, reconstructionLE

Sparse Autoencoders.

Undo the superposition: find many sparse directions whose combinations rebuild the activations.

If a layer stores many sparse features in superposition (plate XXVIII.5), they might be recovered by dictionary learning: find a large set of directions — many more than the layer’s dimension — such that every activation is approximately a sum of a few of them. A sparse autoencoder (SAE) does this with an encoder f = ReLU(W_e x + b_e) and a decoder x̂ = W_d f, trained to reconstruct x while an L1 penalty on f keeps most features at zero. Anthropic (2023, 2024) and others trained SAEs on large models and found many features that fire on a single interpretable thing, from DNA sequences to code errors to specific landmarks.

The course model micro (four layers, d = 128; plate I.9) is opened up in the page. The workshop trains an SAE with 256 features (twice the layer’s width) on the residual stream after micro’s third layer, over about 2 900 tokens of held-out English and Chinese course text, and shows how much variance it reconstructs, how many features are active per token, and the tokens that most strongly activate some of its features.

Below: reconstruction and sparsity for the chosen L1 penalty, and for six frequently used features the tokens that activate them most.

What you will learn
Workshop
SAE on micro’s residual stream after layer 3 (standardised), 256 features with unit-norm decoder columns, trained for 1 200 steps of Adam on batches of 64 tokens. Variance explained and active features per token measured over all tokens; a feature is dead if it never activates. Top tokens: the six tokens with the largest activation of each feature.

Reconstruction against sparsity

With λ = 1 the SAE explains 86.9% of the variance of micro’s layer-3 activations using 40.8 of its 256 features per token; with λ = 0.1 it explains 97.5% but uses 145 features per token — barely sparse — and with λ = 3 it uses 17.8 and explains 69.9%. The penalty sets where on the trade-off the dictionary sits, and only the sparser settings give features worth reading: some of these fire on a single word, on a family of Chinese punctuation, on a script.

Large-scale SAEs have made interpretability concrete — features can be inspected, labelled by what activates them, and used to steer behaviour (plate XXVIII.7) — and they also have known limits. Reconstruction is never perfect, and the missing variance can matter to behaviour; features split and merge as the dictionary grows; some are dead or fire densely; and a feature’s label, inferred from its top examples, can miss what it does elsewhere. SAE features are a useful basis for looking at a model, not yet a complete account of it.

Anthropic’s 2024 SAE on Claude 3 Sonnet had up to 34 million features; one fired on mentions of the Golden Gate Bridge, and amplifying it made the model talk about the bridge in every answer.

Terms

Sparse autoencoder
An autoencoder trained to rebuild activations from a few active features.
Dictionary
The set of decoder directions an SAE learns.
L0
The number of features active for a token.
Dead feature
A feature that never activates.
Sparse autoencoders rewrite activations as a few dictionary directions; the L1 penalty trades reconstruction for interpretable sparsity.

Sources

  1. Bricken et al., “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning”, Anthropic, 2023.
  2. Templeton et al., “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet”, Anthropic, 2024.
  3. Cunningham et al., “Sparse Autoencoders Find Highly Interpretable Features in Language Models”, ICLR 2024.