Vol. XXVI · Training Agents Plate 8 of 10RL infrastructure, asynchronous RL, rollout workers, staleness, off-policy, throughput, stragglersCE

RL Infrastructure.

Never let the slowest episode set the pace — but know how old your data is.

An RL step has two halves: rollout workers sample episodes with the current policy — for agents, running tools, waiting for test suites or web pages — and a trainer computes the update. Episode lengths vary enormously: most coding attempts finish quickly, a few run long. In a synchronous loop the trainer waits for all B episodes of a batch, so every step lasts as long as its slowest episode, and while the trainer updates, the workers wait too.

Asynchronous RL decouples them: workers run continuously, the trainer takes the first B episodes to finish, and new weights are pushed to workers when ready. Throughput rises, but an episode may have started under a policy one or more updates old — stale data — which the importance-ratio clipping of PPO-style losses partly corrects. The workshop simulates W workers for an hour with episode lengths drawn from a log-normal of the chosen median and spread, and an update taking the chosen time.

Below: updates per hour, synchronous and asynchronous, against the spread of episode lengths; readouts give worker utilisation and the staleness distribution.

What you will learn
Workshop
Simulated: 64 rollout workers, batches of 64 episodes, episode durations log-normal with median 20 s and the chosen spread σ (in log units), an update of the chosen duration, one hour. Synchronous: generate 64, wait for all, update, repeat. Asynchronous: workers never stop; each update uses the next 64 to finish. Utilisation: share of worker time spent generating.

Stragglers set the pace

With episode lengths spread by σ = 1 around a 20-second median, the synchronous loop manages 16 updates an hour: each batch waits for its slowest of 64 episodes, and the workers are busy only 14% of the time. The asynchronous loop manages 109, 6.8 times as many, with episodes on average 1.01 updates stale. With narrow spread (σ = 0.3) the gain is 2.6× — asynchrony matters most exactly when episodes are agentic, with long tails.

The cost is off-policy data. An episode generated by an older policy is evidence about that policy, and the PPO-style ratio π_new/π_old with clipping (plate XIX.4) corrects for small gaps; large gaps make updates unreliable, so systems cap staleness, drop very old episodes, or pause and resume long rollouts across updates (“partial rollouts”). Agent RL infrastructure also has to run the environments — sandboxes, browsers, test suites — at scale and reproducibly, which is often a larger engineering effort than the trainer itself.

The same straggler arithmetic appears in distributed training (plate XV.4): any synchronous step lasts as long as its slowest participant.

Terms

Rollout worker
A process that samples episodes with the current policy.
Trainer
The process that computes updates from finished episodes.
Staleness
How many updates old the policy that generated an episode is.
Straggler
The slowest member of a synchronous step, which everyone waits for.
Agent RL is dominated by sampling with long-tailed episodes; asynchronous pipelines keep workers busy at the price of slightly stale, off-policy data.

Sources

  1. Espeholt et al., “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures”, ICML 2018.
  2. Noukhovitch et al., “Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models”, ICLR 2025.
  3. Kimi Team, “Kimi K2”, arXiv:2507.20534, 2025 (partial rollouts for long episodes).