Vol. XXVI · Training Agents Plate 5 of 10loss masking, observations, tool outputs, hallucinated tool results, agent fine-tuning, SFTLE

Training on Actions, Not Observations.

Learn what to do; do not learn to impersonate the world.

A trajectory’s observations are the environment’s text: a calculator’s result, a file’s contents, a web page. If fine-tuning puts a loss on them, the model is trained to predict them — to compute sums it cannot compute, to recite files it has not read. That spends capacity on an impossible imitation, and teaches a habit: given the start of a tool result, write the rest. Masking observations from the loss, so only the model’s own characters are trained, is standard practice in agent fine-tuning for this reason, as masking the prompt is in chat fine-tuning (plate XVII.3).

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop fine-tunes on the same 30 three-number trajectories with the loss on everything, on the whole reply including tool results, or on the model’s actions only, and runs held-out episodes with the calculator and without it — when, after each call, the model must continue by itself.

Below: held-out success with the calculator, success without it, and how often the results the model writes for itself are right, for each masking choice.

What you will learn
Workshop
The character model fine-tuned for 200 steps on 30 demonstrated trajectories with the loss on all characters, on everything after the prompt, or on the model’s own characters only. Held-out: 30 sums, greedy. Without the calculator: after each call the model continues generating, so any result in the transcript is its own; “its own results right” is the share of such results that equal the true sum.

Do not learn to impersonate the tool

With the loss on actions only, the agent solves 77% of held-out sums with the calculator; with the loss on the whole reply, 70%; on everything, including the question, 13%. The difference shows most clearly with the calculator taken away. The actions-only model has never been trained to write a result, so after a call it does not produce one — it wrote 0 results of its own and solves 0%. The model trained on the whole reply writes results itself: 60 of them, 63% right, solving 37% of sums with no tool at all, while looking exactly like an agent reading a calculator.

That is the danger in miniature. A model trained on observations learns to produce plausible observations, and if a tool fails, times out or is absent, it may continue with an invented result in the format of a real one — a fabricated test pass, a made-up search result. Masking observations does not make a model honest, but it stops training from teaching the opposite, and it keeps the gradient on the decisions the model actually makes. The same masking applies in agent RL (plate XXVI.3).

Harnesses add a second defence: tool results arrive in a role or between markers that only the harness can produce, so text the model writes can never be mistaken for a real observation.

Terms

Loss mask
Which tokens contribute to the training loss.
Fabricated observation
A tool result the model wrote itself.
Actions-only loss
Training only on characters the model authored.
Role marker
Harness-controlled markup separating tool output from model output.
Masking observations trains the model on its own decisions; training on tool outputs teaches it to imitate them — and to fabricate them when the tool is missing.

Sources

  1. Chen et al., “FireAct: Toward Language Agent Fine-tuning”, arXiv:2310.05915, 2023.
  2. Zeng et al., “AgentTuning”, arXiv:2310.12823, 2023 (loss on the agent’s outputs only).
  3. Ouyang et al., NeurIPS 2022 (prompt masking in instruction tuning).