Vol. XXVI · Training Agents Plate 1 of 10trajectories, episodes, actions, observations, roles, training data, agentsLE

Trajectories as Data.

One sequence, three authors: the user, the model and the world.

To a language model an agent’s episode is just text: the task, then a call, a result, another call, another result, an answer. But the text has three authors. The prompt comes from the user, the actions from the model, and the observations from tools and the environment — and only the model’s own text is something it should learn to produce. Chat templates and tool-call tokens mark the boundaries so the training code knows which is which.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop lays out one demonstrated trajectory for a three-number sum by author, and counts how many characters each contributes and how many would carry a training loss under each masking choice.

Below: the trajectory for the chosen sum, split by author; the bars give characters per author and the characters that carry loss under each mask.

What you will learn

Read an episode as one sequence.
Identify prompt, actions and observations.
Count what each mask trains on.
Connect trajectories to training choices.

Workshop

A demonstrated trajectory for the chosen held-out sum: the prompt (user), two calculator calls and the answer (model), two results (calculator). Loss characters: those predicted under each mask — all characters after the first, everything after the prompt, or the model’s own characters only.

Whose text is it?

This episode is 31 characters: 11 from the user, 14 written by the model and 6 by the calculator — the tool wrote 30% of the reply. Training on everything after the first character would teach the model to predict all 30, including the user’s question and the calculator’s arithmetic; training on the reply, 20; training on actions only, 14.

Real agent trajectories are dominated by observations — file contents, search results, test output — often by a factor of ten or more over the model’s own text. That makes the choice of what to learn from decisive (plate XXVI.5), makes credit assignment across many turns hard (plate XXVI.4), and makes trajectories expensive to store and replay. Everything in this volume is a decision about which parts of such sequences to imitate, reward or ignore.

A trajectory also records what the environment did, which is what makes it replayable for evaluation and debugging — and why recorded trajectories from a changed environment (a new library version, a different web page) can mislead.

Terms

Trajectory: One episode recorded as a sequence of prompt, actions and observations.
Action: Text the model writes that the environment executes.
Observation: Text the environment writes back.
Role tag: Markup recording which author wrote each part.

An agent trajectory is one sequence with three authors; training must know which characters are the model’s own.

Sources

Yao et al., ReAct, ICLR 2023.
OpenAI, “Harmony response format” and Anthropic, “Tool use” documentation (role-tagged message formats, accessed October 2026).