Vol. XXVII · RL Environments Plate 7 of 11robust graders, hidden tests, property checks, grader design, reward hacking defences, verificationLE

Hardening a Grader.

Make the intended solution the cheapest way to earn the reward.

Plate XXVII.6 located every hack in three conditions — indistinguishable, reachable, easier. Hardening attacks the first two. Distinguish: test where shortcuts fail — edge cases, negative and large inputs, randomised inputs, properties that any correct solution has (a sorted list is a permutation of its input and is ordered). Hide: grade on tests the policy never sees, so memorising the visible ones earns nothing. Protect: run the grader outside the agent’s sandbox, on a copy of the output it cannot alter. A useful check is adversarial: before training, look for the cheapest program that passes, and see whether it is the right one.

The code world of these plates: 40 tasks, each a function f(x) = a·x + b to be written; a policy chooses among four kinds of program — the general formula, a table of the visible tests’ answers, a formula right only for x ≥ 0, and an unrelated constant — starting from preferences that favour the easier ones, and learns by REINFORCE from the share of the grader’s tests its program passes. The workshop trains under four graders: the careless one (three fixed visible inputs from 0 to 9), ten visible inputs from the same range, the three visible plus five hidden inputs drawn from −9 to 9, and visible inputs redrawn from −9 to 9 every episode.

Below: for each grader, the share of general programs the policy ends with and its pass rate on unseen inputs.

What you will learn
Workshop
The code world of these plates: 40 tasks, each a function f(x) = a·x + b to be written; a policy chooses among four kinds of program — the general formula, a table of the visible tests’ answers, a formula right only for x ≥ 0, and an unrelated constant — starting from preferences that favour the easier ones, and learns by REINFORCE from the share of the grader’s tests its program passes. Graders: three fixed inputs from 0–9 (careless); ten fixed inputs from 0–9; the three plus five hidden inputs from −9 to 9 drawn afresh each episode; three visible inputs from −9 to 9 redrawn each episode. Unseen: 20 inputs from −10 to 9. 300 episodes each.

Hidden, fresh, out of reach

Under the careless grader only 12% of the policy’s programs are general and unseen inputs pass 52% of the time. Ten visible tests from the same range change little (5% general, 51% unseen) — more of the same tests share the same blind spot. Five hidden tests on fresh inputs from −9 to 9 change everything: 97% of programs end general and 98% of unseen inputs pass, because neither the half-formula nor a table of visible answers can earn full reward any more. Fresh visible inputs alone reach 57% general: they expose the half-formula, but a table of whatever is shown still passes.

Real graders combine the same moves at scale: hidden test suites, generated inputs and property checks for code; held-out answer keys for maths; final-state checks a script performs outside the agent’s container for computer use. Hardening also means watching for what the designers missed — which is why METR and others publish reward hacks they find, and why teams red-team a new environment with a capable model before training on it, asking it to find the cheapest way to pass.

The hardened grader here also lowers the reward the policy sees early on, because shortcuts stop paying; a lower but honest reward is the sign of a grader doing its job.

Terms

Hidden test
A test the policy never sees, often on fresh inputs.
Property check
A test of a property every correct output has.
Adversarial grader test
Searching for the cheapest output that passes before training.
Hardening
Making the intended solution the easiest way to earn reward.
Harden a grader by testing where shortcuts fail, hiding tests and keeping it out of reach; then search for what it still misses.

Sources

  1. Liu et al., “Is Your Code Generated by ChatGPT Really Correct?” (EvalPlus), NeurIPS 2023.
  2. Claessen and Hughes, “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs”, ICFP 2000 (property-based testing).
  3. METR, “Recent Frontier Models Are Reward Hacking”, 2025.