Vol. XXVII · RL Environments Plate 3 of 11code environments, sandboxes, unit tests, hidden tests, SWE-bench, overfitting to tests, verificationLE

Code Environments.

Tests are a superb grader — for exactly the inputs they try.

Programming is the most successful domain for agent RL because programs can be run: a sandbox — an isolated container with the repository, its dependencies and no access to anything else — executes the agent’s code, and tests decide success. SWE-bench (2024) built 2,294 tasks from real GitHub issues in 12 Python repositories, each graded by the repository’s own tests that the fix must make pass; at its release the best model resolved 2.0% of them.

But a test checks behaviour on the inputs it tries, and nothing else. The code world of these plates: 40 tasks, each a function f(x) = a·x + b to be written; a policy chooses among four kinds of program — the general formula, a table of the visible tests’ answers, a formula right only for x ≥ 0, and an unrelated constant — starting from preferences that favour the easier ones, and learns by REINFORCE from the share of the grader’s tests its program passes. The grader’s tests are inputs from 0 to 9 the policy can see; the workshop also scores each program on 20 inputs from −10 to 9 it never sees.

Below: the policy’s mix of programs over training, and its reward beside its pass rate on the unseen inputs, for the chosen number of visible tests.

What you will learn
Workshop
The code world of these plates: 40 tasks, each a function f(x) = a·x + b to be written; a policy chooses among four kinds of program — the general formula, a table of the visible tests’ answers, a formula right only for x ≥ 0, and an unrelated constant — starting from preferences that favour the easier ones, and learns by REINFORCE from the share of the grader’s tests its program passes. Visible tests: inputs drawn from 0–9 per task, shown to the policy and used by the grader. Unseen: 20 inputs from −10 to 9. REINFORCE with a running baseline, learning rate 0.5, 300 episodes.

Passing the tests is not working

With three visible tests on inputs from 0 to 9, the policy’s reward reaches 99%, while its programs pass only 52% of unseen inputs: it has settled on the formula that is right only for non-negative x (77% of its programs), which passes every visible test and was easier to find than the general one (12%). More visible tests from the same range do not help, because they share the blind spot; the reward is perfect and the behaviour is half wrong.

Real benchmarks meet this in two forms. Tests can be too weak: EvalPlus added many more tests to the HumanEval problems and found that solutions counted as passing failed the stronger suite. And tests can be reachable: an agent in a sandbox that can read or edit the test files, or the grading script, can satisfy them without solving the task. Code environments therefore grade with tests the agent cannot see or change, generated inputs and property checks, and — for issues like SWE-bench’s — the tests that the human fix was written to pass.

A sandbox is also a safety boundary: it lets an agent run arbitrary code while keeping the host, the network and other tasks out of reach — and it must keep the grader out of reach too.

Terms

Sandbox
An isolated environment that runs the agent’s code.
Unit test
A check of a program’s output on chosen inputs.
Hidden test
A test the agent cannot see or change.
Overfitting to tests
Satisfying the visible tests without solving the task.
Tests are exact on the inputs they try and blind elsewhere; a policy rewarded by visible tests learns their blind spots.

Sources

  1. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024.
  2. Chen et al., “Evaluating Large Language Models Trained on Code” (HumanEval), arXiv:2107.03374, 2021.
  3. Liu et al., “Is Your Code Generated by ChatGPT Really Correct?” (EvalPlus), NeurIPS 2023.