Vol. XXVI · Training Agents Plate 6 of 10self-improvement, expert iteration, rejection sampling, STaR, filtering, bootstrapping, agentsLE

Learning From Its Own Attempts.

Try many times, keep what worked, learn from that, try again.

Demonstrations are expensive and RL is delicate; between them sits a simple loop that has trained many agents. Expert iteration (also called rejection-sampling fine-tuning, or STaR for reasoning) samples k attempts at each training task from the current model, keeps the attempts a checker confirms succeeded, fine-tunes on them, and repeats with the improved model. Each round the model sees more successful trajectories of its own making — including for tasks it could only occasionally solve — and the filter keeps its mistakes out of the training data.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop starts from an agent fine-tuned on 30 demonstrations for only 100 steps, which solves few held-out sums, and runs four rounds: 8 attempts per training sum at temperature 1, one success per sum kept, 60 fine-tuning steps on the model’s own actions.

Below: held-out success after each round, with the checker’s filter or keeping every attempt as chosen; the bars give the trajectories kept per round.

What you will learn
Workshop
Start: fine-tuned for 100 steps on 30 demonstrations. Each round: for each of 70 training sums, up to 8 episodes at temperature 1, the first success kept (with the filter) or the first episode kept whatever its result (without); 60 steps of fine-tuning on the kept trajectories, loss on the model’s own characters, learning rate 0.001. Held-out success on 30 sums, greedy.

The filter is the teacher

From 17% on held-out sums, four rounds of keeping only verified successes bring the agent to 47% (its best round reached 47%). The first round found a success for 54 of the 70 training sums within 8 attempts, the last for 64: the model’s own successes grow as it improves, which is what makes the loop climb. Keeping every attempt instead — training on its own output unfiltered — leaves it at 10%: most of what it learns from is its own mistakes.

Expert iteration is reinforcement learning with the simplest possible update: reward 1 trajectories are imitated, reward 0 ones ignored. It lacks RL’s negative signal — nothing pushes failures down — and it can only learn from tasks the model already solves sometimes, so it stalls on tasks with zero pass rate. But it is stable, cheap to run with ordinary fine-tuning code, and its data can be inspected, which is why it is widely used to bootstrap agents before, or alongside, RL. Its guard rail is the grader: a lax one lets mistakes through, and plate XXVII.2 shows how lax graders can be.

The loop also narrows the model: it learns most from the tasks it already half-solves and from its own way of solving them, so diversity of attempts — temperature, several samples — matters for what it can discover.

Terms

Expert iteration
Repeatedly fine-tuning on a model’s own verified successes.
Rejection sampling
Keeping only samples that pass a check.
Bootstrapping
Improving a model from data it generates itself.
Pass rate
The share of attempts at a task that succeed.
Expert iteration imitates the model’s own verified successes; the checker’s filter is what turns self-training into improvement.

Sources

  1. Anthony, Tian and Barber, “Thinking Fast and Slow with Deep Learning and Tree Search” (expert iteration), NeurIPS 2017.
  2. Zelikman et al., “STaR: Bootstrapping Reasoning With Reasoning”, NeurIPS 2022.
  3. Gulcehre et al., “Reinforced Self-Training (ReST) for Language Modeling”, arXiv:2308.08998, 2023.