Vol. XXVI · Training Agents Plate 2 of 10agentic SFT, demonstrations, expert trajectories, behaviour cloning, data quality, imitationLE

Learning From Demonstrations.

Show it how the task is done, enough times and correctly.

Agent training usually starts as chat training did (plate XVII.1): collect trajectories of the task done well — by people, by scripted solvers, or by a stronger model — and fine-tune on them. This behaviour cloning teaches the format of actions, when to call which tool, and the procedure. It is limited the way imitation always is: the model learns to reproduce the demonstrator, including the demonstrator’s mistakes, and only on situations like those it was shown.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop fine-tunes the model on the chosen number of demonstrated trajectories for three-number sums, with the chosen share of them ending in a wrong answer, and measures held-out success.

Below: held-out success against the number of demonstrations, for the chosen share of wrong answers among them; readouts at the chosen number.

What you will learn
Workshop
The character model fine-tuned for 200 steps (AdamW, learning rate 0.002, batches of 16) on the chosen number of demonstrated trajectories from 70 training sums, loss on actions only; wrong demonstrations end with an answer off by 1 to 7 after correct calls. Held-out success on 30 sums, greedy.

Enough, and right

With clean demonstrations, ten trajectories leave the model at 7% on held-out sums — it has not yet learned to copy the right numbers into its calls — while thirty bring it to 77% and seventy to 80%. When a fifth of thirty demonstrations end in a wrong answer, held-out success at thirty falls to 53%; when half do, to 10%. The model copies the demonstrator’s errors as faithfully as its skill: an imitator has no way to know which demonstrations to trust.

Agent SFT data is therefore built with checks: trajectories from strong models are kept only if the task’s grader confirms they succeeded (rejection sampling, plate XIX.6), and human demonstrations are reviewed. Imitation also has a structural limit, the same one that motivated reinforcement learning for reasoning (Vol. XIX): the model is trained on the demonstrator’s states, but at test time its own small errors take it to states no demonstration covered, and nothing in its training says how to recover. That gap is what the next plates close.

The distribution-shift problem of imitation — errors leading to unfamiliar states — was analysed for robot control long before language agents; DAgger (Ross et al., 2011) fixed it by having the expert label the states the learner actually visits.

Terms

Behaviour cloning
Supervised learning of actions from demonstrated trajectories.
Demonstration
A recorded trajectory of the task done well.
Rejection sampling
Keeping only trajectories a grader confirms succeeded.
Distribution shift
Reaching situations at test time that training never showed.
Agent SFT imitates demonstrations, errors included; it needs enough of them, and correct ones, and it cannot teach recovery from states it never showed.

Sources

  1. Pomerleau, “ALVINN: An Autonomous Land Vehicle in a Neural Network”, NeurIPS 1988 (behaviour cloning).
  2. Chen et al., “FireAct: Toward Language Agent Fine-tuning”, arXiv:2310.05915, 2023.
  3. Zeng et al., “AgentTuning: Enabling Generalized Agent Abilities for LLMs”, arXiv:2310.12823, 2023.