Vol. XXVII · RL Environments Plate 1 of 11RL environment, state, actions, observations, reward, termination, episode, interfaceLE

What an Environment Is.

Task in, actions in, observations out, a score at the end. Everything else is design.

Reinforcement learning needs somewhere to act. An environment is a program with a small interface: reset starts an episode and returns the task; step takes the agent’s action and returns an observation, whether the episode has ended, and a reward. Behind the interface it keeps state — a file system, a browser session, a calculator — that actions change. For language agents, actions and observations are text, and the reward comes from a grader that inspects the final state or answer.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. Its environment is ten lines of code: the task is a sum, the actions are calls or an answer, the observations are results, the episode ends at an answer, a malformed call or a full context, and the reward is 1 for the right sum. The workshop runs held-out episodes with the chosen environment settings and lays one out step by step.

Below: one held-out episode as the environment sees it, and success over all held-out episodes for the chosen settings.

What you will learn

Name the parts of an environment.
Read an episode through its interface.
See settings change what happens.
Treat the environment as a design.

Workshop

The tool world with the agent fine-tuned on 30 demonstrations (200 steps). Calculator on: calls are executed; off: nothing answers and the model continues by itself. Faulty: wrong results with probability 30%. Episodes end at an answer, at a malformed call, or when the 32-character context is full. Reward 1 for an exact sum.

Ten lines that decide what is learned

With a reliable calculator the agent solves 77% of held-out sums in 3.00 turns, making 2.00 calls per episode. Make the calculator wrong 30% of the time and the same agent solves 47%; switch it off and the agent — trained only to call — solves 0%. Nothing about the model changed: the environment decides what an action leads to, when an episode ends and what counts as success, and so what any training in it can reward.

Real agent environments are the same interface around far more state: a container with a code repository and a test command, a browser with a set of web sites, a simulated customer with a database. Each design choice — what tools exist, what is observable, how long an episode may run, how the grader decides — becomes part of what the trained model does. The rest of this volume takes those choices one at a time.

The reset–step interface comes from OpenAI Gym (2016), built for game and control environments; agent environments for language models keep it, with text in place of joystick moves and pixels.

Terms

Environment: A program that runs episodes: task, actions, observations, end and reward.
State: What the environment keeps between steps and actions change.
Termination: The rule that ends an episode.
Grader: The part of the environment that computes the reward.

An environment is an interface around state, a termination rule and a grader; each part is a design choice that shapes what the model learns.

Sources

Brockman et al., “OpenAI Gym”, arXiv:1606.01540, 2016 (the reset–step interface).
Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., 2018, chapter 3.