Vol. XXVIII · Interpretability Plate 8 of 9faithfulness, chain of thought, explanations, interventions, monitoring, reasoning modelsLE

Do Explanations Explain?

Change the reasoning and see whether the answer follows.

Reasoning models write out their steps, and those steps are often read as an explanation of how the answer was reached — by users, and by safety monitors that check reasoning for bad intent (plate XX.8). An explanation is faithful if it reflects the process that actually produced the answer. That can fail two ways: the answer may not depend on the stated steps at all (post-hoc rationalisation), or the model may use information it never mentions. Turpin and colleagues (2023) showed models changing answers because of a biased feature in the prompt while giving reasoning that never cited it.

One test is intervention: alter a step of the chain and see whether the answer follows. The workshop uses the chain-of-thought model of plate XIX.7 (the character model fine-tuned on 30 three-number sums with worked chains such as “3+4=7. 7+5=12”). For each held-out sum it forces the chain with the first partial sum changed by 2, lets the model finish, and records whether its answer follows the corrupted chain or the true sum.

Below: how often the final answer follows the corrupted chain, gives the true sum anyway, or something else; and examples.

What you will learn
Workshop
The chain-of-thought model: the character model fine-tuned for 150 steps on 30 three-number sums with worked chains (plate XIX.7). For the 30 held-out sums a+b+c, the chain is forced as “a+b=s̃. s̃+c=” with s̃ = a + b ± 2 (kept a single digit) and the model writes the answer, greedily. Clean: the same with the true partial sum.

Causally involved is not the same as complete

With the true chain, the model answers 90% of held-out sums correctly. With the first partial sum corrupted, its answer follows the corrupted chain 83% of the time and gives the true sum only 3% of the time: this model cannot add three numbers without its chain, so the chain is genuinely part of the computation. For a model this small, the written steps are faithful in the causal sense — change them and the answer changes.

Large reasoning models are not so constrained: they can often reach an answer without the chain, so the chain may be decoration, or may omit what drove the answer. Chen and colleagues (2025) gave reasoning models hints and found they mentioned using the hint in as few as 1% of cases where they used it, often under 20%; outcome-based RL raised faithfulness and then plateaued. Researchers have argued that monitoring chains of thought is a valuable but fragile safety opportunity — fragile because training pressure on the chain can teach models to omit or disguise what they think (plate XXVI.9).

Faithfulness is measured, not assumed: interventions (corrupt a step, cut the chain short, insert a mistake) and hint tests (does the chain admit using a planted clue?) are the two main families of tests.

Terms

Faithful explanation
One that reflects the process actually producing the answer.
Post-hoc rationalisation
Reasoning written to fit an answer reached another way.
Intervention test
Changing a reasoning step to see whether the answer follows.
CoT monitoring
Reading a model’s reasoning to detect unwanted behaviour.
A chain of thought is faithful only if the answer depends on it and it mentions what mattered; test it by intervention and with hints.

Sources

  1. Lanham et al., “Measuring Faithfulness in Chain-of-Thought Reasoning”, arXiv:2307.13702, 2023.
  2. Turpin, Michael, Perez and Bowman, “Language Models Don’t Always Say What They Think”, NeurIPS 2023.
  3. Chen et al., “Reasoning Models Don’t Always Say What They Think”, arXiv:2505.05410, 2025.
  4. Korbak et al., “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety”, arXiv:2507.11473, 2025.