Vol. XIX · Reinforcement Learning for Reasoning Plate 7 of 12chain of thought, reasoning traces, intermediate steps, serial computation, scratchpad, test-time computationAL

Thinking Out Loud.

Two easy sums in a row are easier than one hard one. Writing the steps down lets the model chain what it knows.

A Transformer does a fixed amount of computation per token: one pass through its layers. A problem that needs more serial steps than that cannot be solved in the token that states the answer — but it can be solved across several tokens, if the model writes its intermediate results and reads them back. This is the chain of thought: Wei and colleagues (2022) showed that prompting large models with worked examples made them write reasoning steps and solve far more arithmetic and logic problems; Kojima and colleagues found “let’s think step by step” was enough. Reasoning models are trained to produce such chains at length (plate XIX.10).

The post-training world makes the mechanism visible. Its base model knows sums of two digits. A sum of three numbers, a + b + c with a + b ≤ 9, needs two of them in sequence. The workshop fine-tunes the base, from the same number of examples, to answer such sums directly (“12|”) or with a chain in worksheet form (“3+4=7. 7+5=12|”), and measures held-out accuracy.

Below: held-out accuracy on three-number sums as fine-tuning proceeds, for direct answers or chains of thought, from the chosen number of examples; samples show what the model writes.

What you will learn
Workshop
Live fine-tuning from the recorded base on n three-number sums (a + b ≤ 9, so every partial sum is a digit), 150 steps, loss on answers only, answered directly or with a two-step chain in worksheet form. Held-out accuracy: greedy answers on 30 sums never shown, scored on the final number.

Two easy steps instead of one hard one

From 30 examples, a model taught to write the chain answers 70% of the held-out three-number sums; taught to answer directly, 17%. From 70 examples the gap persists: 90% against 37%. The chain wins because each of its steps — “3+4=7”, then “7+5=12” — is a two-digit sum written exactly as the base model saw thousands of times in its worksheets, while the direct answer asks for a computation pre-training never showed it in one step.

That is the general argument, made small. A chain of thought buys serial computation with tokens, and it works best when each step is something the model can already do; it is also a place for errors, since one wrong step carries forward. Theory agrees: Transformers of bounded depth provably cannot solve some problems in a single output token that they can solve when allowed to write intermediate steps. Whether the written chain is the computation that actually produced the answer — whether it is faithful — is a separate question, taken up in plate XXX.8.

The format here copies the worksheets on purpose. A chain written in a style the model never saw in pre-training would have to be learned from the fine-tuning examples alone — the same dependence on surface form as in plate XVII.1.

Terms

Chain of thought
Intermediate reasoning steps written before the answer.
Serial computation
Computation whose steps depend on earlier ones.
Scratchpad
Tokens a model writes for itself to hold intermediate results.
Faithfulness
Whether a written chain reflects the computation that produced the answer.
A chain of thought buys serial computation with tokens: a hard step becomes several easy ones the model already knows.

Sources

  1. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, NeurIPS 2022.
  2. Kojima et al., “Large Language Models are Zero-Shot Reasoners”, NeurIPS 2022.
  3. Feng et al., “Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective”, NeurIPS 2023.