Drills: Training Agents.
Five questions at a time from the whole volume, until every training choice has its reason.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that agent training is chat training stretched over episodes: imitate good trajectories, then reinforce successful ones — always on the model’s own actions, always against a grader that must be worth optimising.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Explain agent SFT and its limits.
- Reason about multi-turn RL and credit.
- Choose masks, filters and curricula.
- Spot reward hacking.
What to carry forward
A trajectory has three authors and only the model’s actions are trained (plates XXVI.1, XXVI.5); demonstrations teach the procedure, mistakes included (plate XXVI.2); multi-turn RL scores the outcome and updates actions (plate XXVI.3), with credit diluted over long episodes (plate XXVI.4).
Filtered self-training climbs where unfiltered does not (plate XXVI.6); long horizons starve group-relative RL (plate XXVI.7); asynchronous pipelines beat stragglers at the cost of staleness (plate XXVI.8); and a rising reward can be a hack (plate XXVI.9). Vol. XXVII builds the environments these methods train in.
Terms
- Behaviour cloning
- Imitating demonstrations.
- Expert iteration
- Fine-tuning on verified successes.
- Dead group
- Attempts with equal rewards.
- Staleness
- Updates since an episode’s policy.