Vol. XXVIII · Interpretability Plate 5 of 9superposition, features, polysemanticity, sparsity, toy models, interference, representationLE

Superposition.

More features than dimensions — if they rarely appear together.

If each neuron of a model stood for one concept, interpretation would be easy. They usually do not: single neurons respond to several unrelated things (polysemanticity). Elhage and colleagues (2022) proposed an explanation with a toy model: when a network has to represent more features than it has dimensions, and the features are sparse — each is usually zero — it can store them in superposition, as directions that are not orthogonal, accepting some interference when two happen to be active together because that is rare.

The workshop trains their toy model: 6 features of decreasing importance, each zero with probability S and otherwise uniform on [0, 1], are mapped into m = 2 dimensions by W, then back by Wᵀ with a ReLU, minimising importance-weighted reconstruction error. It shows the learned direction of each feature in the two dimensions and how many are represented.

Below: each feature’s learned direction in the two hidden dimensions (longer means better represented), and the length of each, for the chosen sparsity.

What you will learn
Workshop
6 features, importance 0.8^i, mapped into 2 dimensions; trained for 3 000 steps of Adam (learning rate 0.01, batches of 256). Represented: a feature whose direction has length above 0.5. Arrows from the origin to each feature’s point; labels give the feature number (1 most important).

Interference is cheap when features are rare

With dense features (S = 0) the model represents 2 of the 6 — the two most important, on orthogonal axes — and ignores the rest, exactly as principal components would. At sparsity 0.7 it represents 4, packing them at angles so that each pair interferes only when both are active; at 0.97 it represents 5. The geometry it finds — antipodal pairs, pentagons — follows from the trade between the value of each extra feature and the cost of interference.

If real models do this at scale, neurons will be polysemantic by construction, and the natural units of analysis are directions, not neurons. That is the motivation for sparse autoencoders (next plate), which try to recover the many sparse directions a layer superposes. Superposition also explains why models are hard to edit locally — changing one direction disturbs the features that share its space — and why interpretability results on toy models transfer to real ones only with care.

Superposition is the representational counterpart of compressed sensing: sparse signals can be recovered from fewer measurements than their dimension, as long as they are sparse enough.

Terms

Superposition
Representing more features than dimensions with non-orthogonal directions.
Polysemantic neuron
A neuron responding to several unrelated features.
Feature sparsity
How rarely a feature is active.
Interference
Error when two superposed features are active together.
Sparse features can be packed into fewer dimensions than their number; that makes neurons polysemantic and directions the unit of analysis.

Sources

  1. Elhage et al., “Toy Models of Superposition”, Anthropic, 2022.
  2. Olah et al., “Zoom In: An Introduction to Circuits”, Distill, 2020 (polysemantic neurons).
  3. Arora, Li, Liang, Ma and Risteski, “Linear Algebraic Structure of Word Senses”, TACL 2018.