Vol. XXVII · RL Environments Plate 8 of 11environment diversity, generalisation, transfer, multi-task training, overfitting to an environmentLE

Breadth of Environments.

Train on one world and you learn that world.

RL and fine-tuning teach what the environment rewards, in the form it presents. A policy trained to fix bugs in one repository learns that repository’s conventions; a policy trained on one website’s layout learns the layout. Skills transfer to new environments only as far as the training environments varied along the dimensions that change. That is why frontier agent training uses thousands of environments — many repositories, many tools, many task families — and why their number and variety are themselves an object of engineering.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. It has two task families with different formats: two two-digit numbers (one call) and three digits (two calls). The workshop fine-tunes on demonstrations from one family, the other, or a mix, with the same 70 demonstrations and 400 steps, and tests on held-out sums of both.

Below: held-out success on each task family for the chosen training mix.

What you will learn
Workshop
The character model fine-tuned for 400 steps on 70 demonstrations from the two-digit family, the three-digit family, or a mix of 35 and 35, loss on actions only. Held-out success on 30 sums of each family, greedy.

Variety is what transfers

Trained only on three-digit sums, the agent solves 93% of them and 0% of two-digit sums: it has never written a two-digit operand into a call. Trained only on two-digit sums, it solves 100% of those and 0% of three-digit sums: it has never made a second call. Trained on both, with the same number of demonstrations split between them, it solves 30% and 83%. Each environment taught its own procedure; only the mixture taught both.

The lesson scales. Cobbe and colleagues showed game-playing agents overfitting to the specific levels they trained on until the number of distinct levels ran into thousands; language agents trained in a few environments learn those environments’ quirks. Agent training pipelines therefore invest in breadth — many repositories and languages for coding, many tools and schemas for tool use, synthetic environments generated at scale — and hold whole environments out for evaluation, not just tasks (plate XXVII.9).

Mixing has a cost here: with the demonstrations split in half, each family is learned less well than with all 70 devoted to it. Breadth trades depth per environment for coverage across them.

Terms

Environment diversity
Variety in the environments a policy trains in.
Transfer
Performance in an environment not trained on.
Held-out environment
A whole environment kept out of training for evaluation.
Overfitting to an environment
Learning its quirks rather than the general skill.
Policies learn the environments they train in; breadth of environments, not just of tasks, is what transfers.

Sources

  1. Cobbe et al., “Quantifying Generalization in Reinforcement Learning” (CoinRun), ICML 2019.
  2. Kimi Team, “Kimi K2”, arXiv:2507.20534, 2025 (large-scale synthesis of tool-use environments).
  3. Pan et al., “SWE-Gym”, arXiv:2412.21139, 2024.