Vol. XXVIII · Interpretability Plate 2 of 9linear probes, representation, control tasks, selectivity, residual stream, decodabilityAE

Linear Probes.

If a straight line can read it from the activations, the information is there — in some form.

To ask what a layer represents, train a probe: a small, usually linear, classifier that predicts some property from the activations at that layer, tested on held-out examples. High probe accuracy shows the property is decodable there. It does not show the model uses it, and a flexible probe can learn properties from almost anything: Hewitt and Liang (2019) proposed control tasks — labels assigned at random to each word type — that a probe can only solve by memorising token identity, and measure selectivity as accuracy on the real task minus accuracy on the control.

The course model micro (four layers, d = 128; plate I.9) is opened up in the page. The real task is whether the next token begins a new word — something the model must anticipate from context. The control gives every token type a fixed random label. A logistic-regression probe is trained on 70% of positions of 705 tokens of held-out course text, English and Chinese at each depth and tested on the rest.

Below: probe accuracy at each depth for the real task and the control, against the majority baseline of each.

What you will learn
Workshop
Logistic regression on standardised residual vectors (128 dimensions) at each depth, trained by gradient descent (150 iterations, light L2), 70% of positions for training and 30% for testing. Task: the next token begins with a space or is a Chinese character. Control: a fixed random bit per token type.

Decodable is not the same as used

Whether the next token starts a new word is readable at the embedding about as often as guessing the majority class (69% against 70%); by depth 2 a linear probe reads it 82% of the time — the model has computed something about what comes next. The control task goes the other way: at the embedding, where each token is still mostly its own identity, the probe memorises the random labels to 72% (baseline 54%); by the final depth only 65%, as the stream comes to encode context rather than the token itself.

Probes are cheap and widely used, and their results need care in both directions. A successful probe shows information is present and linearly arranged, but the model may never read it; a failed linear probe does not show absence, only that the information is not linear. The standard fixes are controls like the one here, comparisons across layers and models, and causal tests: if the model uses a direction, intervening on it should change its behaviour — the subject of the next plate.

Probing has found linear representations of syntax, sentiment, truthfulness and the board state of games in language models — each a claim about decodability that later work tested causally.

Terms

Probe
A classifier trained to read a property from activations.
Control task
Random labels per word type, solvable only by memorising identity.
Selectivity
Task accuracy minus control accuracy.
Decodable
Recoverable from activations by a given probe.
A probe shows what is decodable, not what is used; controls and causal tests are what turn a probe result into an explanation.

Sources

  1. Alain and Bengio, “Understanding intermediate layers using linear classifier probes”, arXiv:1610.01644, 2016.
  2. Hewitt and Liang, “Designing and Interpreting Probes with Control Tasks”, EMNLP 2019.
  3. Belinkov, “Probing Classifiers: Promises, Shortcomings, and Advances”, Computational Linguistics, 2022.