Vol. XXVII · RL Environments Plate 4 of 11task generation, templates, difficulty, pass rate, synthetic tasks, RL data, mutationLE

Generating Tasks.

The useful tasks are the ones the policy sometimes gets right.

Environments need tasks: issues to fix, questions to research, sums to compute. Collected tasks run out, so most RL pipelines generate them — from templates with parameters (numbers, file names, constraints), by mutating existing tasks (change a requirement, insert a bug), or by asking a model to write new ones that a checker can verify. What makes a task useful is not only that it is correct and gradable but that its difficulty suits the policy: the GRPO signal comes only from tasks whose attempts sometimes succeed and sometimes fail (plate XXVI.7).

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. Its tasks come from two templates — sums of two two-digit numbers and sums of three digits — each a family of 100 generated items. The workshop measures the pass rate of each of the 70 training sums of three digits, over 8 attempts at temperature 1, for an agent fine-tuned for the chosen number of steps.

Below: the distribution of per-task pass rates for the chosen agent; readouts give the shares of tasks never solved, always solved and in between.

What you will learn
Workshop
70 training sums of three digits (the first two summing to at least 10). Pass rate: successes in 8 episodes at temperature 1. The agent fine-tuned on 30 demonstrations for the chosen number of steps.

Difficulty relative to the policy

For the weak agent, 27% of training sums are never solved in 8 attempts and 73% fall in the learnable band: this task set suits it. After 200 steps of fine-tuning the picture shifts: 66% of sums are always solved — just as useless for GRPO as never — and only 29% remain in the band. The same task set suits one policy and is too easy for the next, which is why task difficulty must be measured against the current policy, not fixed once, and why generators must keep producing harder tasks as training succeeds.

Generators therefore expose difficulty knobs — number of steps, size of numbers, distractors, required tools — and pipelines filter or reweight tasks by measured pass rate, generating more of the band as the policy moves (plate XXVII.5). Generated tasks bring their own risks: a template repeats its structure, so a policy can learn the template rather than the skill (plate XXVII.8); a model-written task can be wrong or ungradable; and anything generated from a test set leaks it (plate XXVII.9).

Measuring pass rates costs rollouts too: estimating each task’s difficulty with 8 attempts is 8 episodes per task before any training happens, so pipelines reuse the rollouts of training itself to update their estimates.

Terms

Template
A parameterised pattern that generates many tasks.
Mutation
Changing an existing task to make a new one.
Pass rate
The share of attempts at a task that succeed.
Learnable band
Tasks with pass rate strictly between 0 and 1.
A task’s value depends on its pass rate for the current policy; generators need difficulty knobs and pipelines need to measure where each task sits.

Sources

  1. Zelikman et al., “STaR”, NeurIPS 2022.
  2. Yu et al., “DAPO”, arXiv:2503.14476, 2025.
  3. Pan et al., “Training Software Engineering Agents and Verifiers with SWE-Gym”, arXiv:2412.21139, 2024.