Vol. XXXI · History and Projects Plate 11 of 14project, RL environment, grader, hidden tests, reward hacking, specification gaming, codingLE

Project: Build an Environment.

The policy learns whatever the grader pays for. Design the grader.

The code world of these plates: 40 tasks, each a function f(x) = a·x + b to be written; a policy chooses among four kinds of program — the general formula, a table of the visible tests’ answers, a formula right only for x ≥ 0, and an unrelated constant — starting from preferences that favour the easier ones, and learns by REINFORCE from the share of the grader’s tests its program passes.

The brief: after training, the policy puts at least 0.90 of its probability on the general program and passes at least 0.90 of fresh tests it never saw, with no more than ten tests run per episode, since each costs sandbox time. Three levers: how many tests the policy sees, how many hidden tests the grader adds, and whether the visible inputs are redrawn every episode. The starting grader shows 3 tests, hides none, and keeps the inputs fixed: the policy ends with 0.12 on the general program. A project plate grades the design against a brief: every check below the workshop must pass, and a passing design is saved in this browser.

Left: the policy’s mix of program kinds as training proceeds. Right: the general program’s share and the pass rate on unseen tests against the brief.

What you will learn
Workshop
The code world of plates XXVII.3 and XXVII.6, REINFORCE with a running baseline, 300 steps at learning rate 0.5. Unseen-test pass rate is measured on fresh inputs from −9 to 9.

A grader is a specification

With only the visible tests, the starting grader pays the table of their answers, and the formula right only for x ≥ 0, as fully as the general program — so the policy drifts to whichever was easier to start with, collecting a training reward of 0.99 while passing only 0.52 of unseen tests. Every reward was earned; none of it was for the intended behaviour. More visible tests do not help: whatever the policy can see, it can memorise.

Tests the policy cannot see make the shortcuts unprofitable, and a few suffice when they are drawn fresh. Redrawing the visible inputs is a cheaper remedy for tables, but not for formulas that are right on part of the input. Real environments use all of these and more — held-out tests, randomised instances, checks on the solution’s form — and still find policies exploiting gaps nobody listed; the pass rate on tests nobody trained against is the measurement that catches them.

In a toy this small, learning itself is noisy: a few designs that should work end in a poor mix, so the brief judges the trained result, not the design’s intention.

Terms

Grader
The code that turns a policy’s output into a reward.
Hidden tests
Tests the grader runs that the policy never sees.
Specification gaming
Earning reward by a behaviour the designer did not intend.
Unseen-test pass rate
Success on fresh tests, the measure of the intended behaviour.
An environment teaches what its grader pays for, so pay only for the intended solution — with tests the policy cannot see — and measure on tests nobody trained against.

Sources

  1. Krakovna et al., “Specification gaming: the flip side of AI ingenuity”, DeepMind blog, 2020.
  2. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024.
  3. Baker et al., “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation”, arXiv:2503.11926, 2025.