Drills: Environments.
Five questions at a time from the whole volume, until every environment design choice has its failure mode attached.
This plate draws rounds of five from the questions and predictions of the ten plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that the environment is half of what RL trains: its tasks set what can be learned, its grader what will be learned, its breadth what will transfer, and its separation from evaluation whether anyone can tell.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Describe environment interfaces and graders.
- Choose and schedule tasks.
- Find and fix reward hacks.
- Keep evaluation honest.
What to carry forward
An environment is reset, step, state, termination and a grader (plate XXVII.1); graders’ false positives are rewards for mistakes (plate XXVII.2); tests are blind where they do not look (plate XXVII.3); task value depends on pass rate (plate XXVII.4), and curricula keep training in the band (plate XXVII.5).
Hacks need indistinguishable, reachable, easier behaviour (plate XXVII.6) and are fixed by distinguishing, hiding and protecting (plate XXVII.7); breadth of environments transfers (plate XXVII.8); leaks inflate scores (plate XXVII.9); and simulators are learned models of people (plate XXVII.10). Part Seven turns to what is inside models and where the field stands.
Terms
- Grader
- The reward’s source.
- Hidden test
- A test the policy cannot see.
- Curriculum
- Task selection by pass rate.
- Leakage
- Test material in training.