Vol. XXVI · Training Agents Plate 3 of 10multi-turn RL, agent RL, GRPO, outcome reward, episodes, tool results, policy gradientLE

Multi-Turn RL.

Run the whole episode, score the end, and push up every action of the episodes that scored well.

Reinforcement learning extends to agents with one change: an episode is not one reply but a sequence of turns, with the environment’s observations inserted between the model’s actions. The model samples actions; the harness runs the tools and appends their results; at the end a grader scores the outcome. GRPO (plate XIX.5) then compares the episodes sampled for the same task: each one’s advantage is its reward relative to the group, applied to every character the model wrote — and to none of the characters the environment wrote, which were not the model’s choices.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop starts from the agent fine-tuned on 30 demonstrations and runs GRPO on the 70 training sums — 8 sums per step, 8 episodes per sum, sampled at temperature 1 — rewarding an exact final answer, with a reliable calculator or one that is wrong 30% of the time.

Below: held-out success during training and the mean reward of the sampled episodes; readouts give the start and end.

What you will learn
Workshop
GRPO from the agent fine-tuned on 30 demonstrations (200 steps): 40 steps of 8 training sums × 8 episodes at temperature 1, learning rate 0.0003, clipped ratio, loss on the model’s own characters. Reward 1 for an exact final answer. Held-out success on 30 sums, greedy, with a reliable calculator. A faulty calculator returns a wrong result 30% of the time during training.

The same algorithm, a longer episode

From the fine-tuned agent at 77% on held-out sums, 40 GRPO steps bring it to 83%; the sampled episodes, drawn at temperature 1 and so noisier than greedy decoding, earn 81% of the reward by the end. With a calculator that is wrong 30% of the time during training, the reward the policy sees is capped well below — 50% — because no action can fix the tool’s error, and the held-out result moves to 83%. Noise in the environment is noise in the gradient.

Large agentic RL runs follow this pattern at vastly greater scale: episodes of hundreds of turns in code repositories, browsers or terminals, graded by tests or task checkers, with the same masking of observations and the same group-relative advantages. The difficulties grow with the episode: rewards become sparser (plate XXVI.7), credit for one good decision is spread over every action (plate XXVI.4), and sampling episodes becomes the dominant cost of training (plate XXVI.8).

The environment is part of the training system: its speed limits the sampling rate, its bugs become rewards, and its randomness enters every gradient. Building it well is the subject of Vol. XXVII.

Terms

Multi-turn RL
Reinforcement learning over episodes of several actions and observations.
Outcome reward
A score given only at the end of the episode.
Group-relative advantage
An episode’s reward compared with others for the same task.
Observation masking
Applying the policy gradient only to the model’s own tokens.
Multi-turn RL samples whole episodes, scores the outcome and updates the model’s own actions; a noisy environment caps what it can learn from.

Sources

  1. Shao et al., “DeepSeekMath” (GRPO), arXiv:2402.03300, 2024.
  2. Zhou et al., “ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL”, ICML 2024.
  3. Kimi Team, “Kimi K2: Open Agentic Intelligence”, arXiv:2507.20534, 2025 (large-scale agentic RL).