Vol. XXVII · RL Environments Plate 9 of 11train-test separation, leakage, contamination, held-out environments, honest evaluation, agentsLE

Training and Test Environments.

A score on tasks you trained on is a memory test.

Evaluation of agents inherits every pitfall of benchmark evaluation (Vol. XXI) and adds its own. Tasks generated by the same template as training tasks share their structure; environments shared between training and evaluation let a policy learn their quirks; and test tasks themselves can leak into training through generated data, through public repositories that contain their solutions, or simply by mistake. A leaked task is solved by recall, and a score that includes it overstates skill (plate XXI.7).

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop fine-tunes the agent on its 30 demonstrations plus, optionally, demonstrations of the first ten of its 30 held-out sums, and reports success on the leaked ten and the clean twenty separately.

Below: held-out success on the leaked and clean held-out sums, and on all thirty together, with or without the leak.

What you will learn

Define leakage in agent evaluation.
Measure its effect on a score.
Separate leaked from clean items.
Hold out environments, not just tasks.

Workshop

The character model fine-tuned for 200 steps on 30 demonstrations of three-digit sums, plus demonstrations of the first 10 held-out sums when leaked; loss on actions only. Success on the 10 leaked and 20 clean held-out sums, greedy.

Memory dressed as skill

With ten test sums leaked into its demonstrations, the agent solves 100% of those ten and 70% of the twenty clean ones; its score on all thirty is 80%, against 77% for the same training without the leak. The headline moved by 3% with no new ability — and a reader seeing only the headline could not tell. Splitting the score by whether items could have been seen is the check, and it requires knowing what training contained.

Agent evaluations add structural leaks: an environment whose tasks share templates with training tasks; a repository whose later commits contain the fix; a web site the agent browsed in training. Kapoor and colleagues documented agent benchmarks whose held-out sets were too close to training to measure generality, and argued for holding out whole environments and reporting the held-out level explicitly. Honest measurement of agents means treating environments, not just tasks, as the unit of separation.

For agents with web access there is a further leak: the answer may be online. Some benchmarks now block their own solutions from the agent’s browser, or check that the agent did not simply find them.

Terms

Leakage: Test material appearing in training.
Clean split: Test items verified absent from training.
Held-out environment: An environment used only for evaluation.
Structural leak: Shared templates, repositories or sites between training and test.

A leaked test task measures memory, not skill; split scores by exposure and hold out whole environments.

Sources

Jimenez et al., SWE-bench, ICLR 2024 (tasks from public repositories with known fixes).
Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic” (GSM1k), NeurIPS 2024.
Kapoor et al., “AI Agents That Matter”, arXiv:2407.01502, 2024.