Vol. XXVII · RL Environments Plate 5 of 11curriculum learning, pass-rate targeting, dead groups, task selection, GRPO, adaptive samplingLE

Curricula.

Train on what the policy can almost do.

People learn arithmetic before algebra; Bengio and colleagues (2009) showed neural networks also learn faster on examples ordered from easy to hard — a curriculum. In RL for language models the useful form is adaptive: estimate each task’s pass rate under the current policy and sample training tasks from the band where it is neither 0 nor 1, so that every group has something to compare. As the policy improves, the band moves to harder tasks by itself.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop takes the measured pass rate of each of its 70 training sums (8 attempts each, as in plate XXVII.4) and computes, for a group of the chosen size, the chance that every attempt in it gets the same reward — a dead group — when tasks are drawn uniformly or only from the chosen band.

Below: the training tasks by measured pass rate, with the band marked; readouts give the expected share of dead groups under uniform and banded sampling.

What you will learn

Define a curriculum.
Describe pass-rate targeting.
Compute dead groups with and without it.
See the band move with the policy.

Workshop

Pass rates of the 70 training sums of three digits, each from 8 episodes at temperature 1, for the agent fine-tuned on 30 demonstrations for the chosen number of steps. Expected dead share: the mean of p^G + (1 − p)^G over the tasks sampled, taking the measured rate as p. Band: tasks with measured rate between the chosen bounds.

Spend rollouts where they teach

For the weak agent with groups of 8, drawing training sums uniformly gives 39% dead groups in expectation — mostly sums it never solves — and drawing only from the band of pass rates between 1/8 and 7/8 (51 sums) gives 17%. For the stronger agent the dead groups come from the other end, sums it always solves: 80% uniform against 30% in the band of its 20 sums. The same band rule serves both, because it is defined by the policy’s current pass rates, not by fixed difficulty.

In a real run the pass rates are estimated from the training rollouts themselves and change as the policy learns, so the band is refreshed continually; DAPO-style dynamic sampling applies the same idea per batch by discarding groups that came back all-pass or all-fail and drawing more. The cost is bookkeeping and some bias — training concentrates on a moving band and may neglect tasks it once solved, so evaluation must watch for forgetting. In this tiny world the effect of the curriculum on final accuracy is within the noise of a 40-step run; its effect on wasted rollouts is not.

The band never contains a task the policy fails every time; whatever it is to learn there must come from transfer — which is why curricula are paired with generators that can make slightly easier variants of hard tasks.

Terms

Curriculum: An ordering or selection of training tasks by difficulty.
Pass-rate targeting: Sampling tasks whose current pass rate is in a middle band.
Dead group: A group of attempts with identical rewards and zero advantage.
Forgetting: Losing skill on tasks no longer trained on.

An adaptive curriculum keeps training on tasks the policy can almost do, so fewer groups are dead and each rollout can teach.

Sources

Bengio, Louradour, Collobert and Weston, “Curriculum Learning”, ICML 2009.
Yu et al., “DAPO”, arXiv:2503.14476, 2025 (dynamic sampling).
Team OLMo et al., “2 OLMo 2 Furious”, arXiv:2501.00656, 2025 (RL with verifiable rewards and data filtering).