Vol. XIX · Reinforcement Learning for Reasoning Plate 11 of 12distillation, reasoning traces, chain of thought, rejection sampling, student, teacher, DeepSeek-R1 distillationLE

Distilling Reasoning.

Copy the teacher’s working, not just its answers — and keep only the working that came out right.

RL on a large model is expensive; copying its behaviour into a smaller one is not. DeepSeek-R1’s authors fine-tuned several smaller open models on about 800 000 samples generated by R1 — reasoning traces filtered for correctness, plus general data — and reported that the distilled small models outperformed the same small models trained with RL directly. Much of the open ecosystem of small reasoning models is built this way: a strong teacher’s chains of thought, filtered, become a student’s fine-tuning data.

In the post-training world the teacher is the chain-of-thought model of plate XIX.7, trained on 30 examples. It answers the 70 training sums; a student is then fine-tuned from the base on the teacher’s final answers alone, on its full traces, or on the traces whose final answer the verifier accepts.

Below: the student’s held-out accuracy as fine-tuning proceeds, for the chosen kind of teacher data, beside the teacher’s own accuracy.

What you will learn

Describe distillation of reasoning traces.
Show that traces carry more than answers.
Show filtering lifting the student above the teacher.
Explain why distillation is cheaper than RL.

Workshop

Teacher: the base fine-tuned on 30 three-number sums with chains of thought (plate XIX.7). It answers the 70 training sums greedily. Student: fine-tuned from the base for 200 steps on the teacher’s final answers only (direct format), on its full traces, or on the traces with a verified final answer. Accuracy on the 30 held-out sums.

Copy the working, keep the right ones

The teacher answers 70% of the held-out sums and 84% of the training sums. A student given only its final answers reaches 27%: it must find the three-number sum in one step, as in plate XIX.7. Given the teacher’s full traces, wrong ones included, it reaches 80%. Given only the 59 traces whose answer the verifier accepts, it reaches 100% — better than the teacher, because it learns the teacher’s method without the teacher’s slips.

The three ingredients of the result are the three of real reasoning distillation: traces carry the method, not just the answer; a verifier removes the teacher’s errors; and the student starts from a base that can execute each step. Distillation is far cheaper than RL — one generation pass of the teacher and ordinary fine-tuning — which is why small reasoning models are mostly distilled. Its limit is the teacher: a student learns to reason the way its teacher does, and gains what RL would have discovered only if the teacher discovered it first.

The s1 study found that about 1 000 carefully chosen reasoning traces from a strong teacher were enough to give a 32B model much of the teacher’s reasoning style — another instance of fine-tuning’s small appetite for well-chosen data (plate XVII.5).

Terms

Reasoning trace: The chain of thought a model wrote on its way to an answer.
Trace distillation: Fine-tuning a student on a teacher’s reasoning traces.
Correctness filter: Keeping only traces whose final answer is verified.
Student ceiling: The limit a teacher’s method places on what a student learns.

Distil the teacher’s working, filtered for correctness: traces teach the method, the filter removes the slips, and the student can pass the teacher.

Sources

DeepSeek-AI, “DeepSeek-R1”, arXiv:2501.12948, 2025, sections 2.4 and 4.1 (distillation into Qwen and Llama models).
Muennighoff et al., “s1: Simple test-time scaling”, arXiv:2501.19393, 2025.
Zelikman et al., “STaR”, NeurIPS 2022.