Vol. XXVI · Training Agents Plate 7 of 10long horizon, sparse reward, dead groups, GRPO, curriculum, sub-goals, agent trainingCE

Long Horizons.

If no attempt in the group succeeds, the group teaches nothing.

Group-relative RL (plate XIX.5) learns only from groups whose episodes differ in reward: if all G attempts at a task fail, every advantage is zero. For an agent whose per-step success is p, an n-step task succeeds with probability about pⁿ (plate XXV.5), and the chance that at least one of G attempts succeeds is 1 − (1 − pⁿ)^G. As n grows, that chance collapses, the share of dead groups rises, and compute spent sampling long episodes buys no gradient. Long tasks are expensive per episode and uninformative per episode at once.

The remedies are about keeping tasks in the learnable band: start from a policy that already succeeds sometimes (plate XXVI.2); choose tasks whose pass rate is neither 0 nor 1 (plate XXVII.5); break long tasks into stages with checked sub-goals; and give the agent context management so that a long task does not fail merely because its window filled (plate XXV.4). The workshop computes the useful share of groups for the chosen per-step success, task length and group size.

Below: the share of groups with at least one success, and with a mix of successes and failures, against task length; readouts at the chosen length.

What you will learn

Explain dead groups in group-relative RL.
Compute the useful share of groups.
See long tasks starve learning.
Name ways to keep tasks learnable.

Workshop

Independent attempts; a task of n steps succeeds with probability pⁿ. A group is useful when its attempts differ in reward — at least one success and at least one failure. Rollout cost: n steps per attempt, G attempts per task; useful groups per million steps sampled.

Long tasks starve the gradient

At 98% per step and 8 attempts, 68% of groups for a 100-step task carry a learning signal; at 300 steps, 1.9% — almost every group is all failures. Sampling 64 attempts instead of 8 recovers some (14% at 300 steps) at eight times the cost, while raising per-step success to 99.5% recovers more (87%). Reliability per step is worth more than attempts per task, because it enters the exponent.

Training recipes for long-horizon agents are largely about staying in the band where groups are mixed. DAPO-style dynamic sampling discards all-pass and all-fail groups and samples more tasks to refill the batch; curricula schedule tasks by pass rate (plate XXVII.5); and tasks are decomposed so that each training episode is short enough to sometimes succeed, while context management and memory let the agent carry long tasks through without the horizon itself being trained end to end at once.

Dead groups waste more than the update: the episodes still had to be sampled, and for agents sampling is most of the cost (plate XXVI.8).

Terms

Dead group: A group of attempts with identical rewards, giving zero advantage.
Horizon: The number of steps a task takes.
Dynamic sampling: Discarding uninformative groups and refilling the batch.
Sub-goal: An intermediate checked milestone within a long task.

Long tasks make successful attempts rare and groups uninformative; per-step reliability, curricula and decomposition keep RL in the learnable band.

Sources

Shao et al., “DeepSeekMath” (GRPO), arXiv:2402.03300, 2024.
Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale”, arXiv:2503.14476, 2025 (dynamic sampling to drop all-correct and all-wrong groups).