Vol. XXV · Tools and Agents Plate 2 of 9agent loop, ReAct, observations, actions, multi-step, tool results, unreliable toolsLE

The Agent Loop.

Act, observe, act again on what you saw. Repeat until done.

An agent is a model in a loop: it reads the context, writes an action (a tool call, a command, a message), the environment executes it and appends an observation, and the model goes round again until it decides to stop. ReAct (Yao et al., 2023) gave the pattern its name by interleaving written reasoning with actions; modern coding and research agents run the same loop for hundreds of turns. What distinguishes the loop from a single call is that later actions depend on earlier observations.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. Here the task is a three-number sum such as 7 + 8 + 9: the model must call the calculator on the first two numbers, read the result, and call it again with that result and the third number. The workshop fine-tunes on 30 demonstrated trajectories and runs held-out episodes with a calculator that is right always or only most of the time.

Below: held-out episodes with two calls each; the bars give held-out success as the calculator’s error rate rises.

What you will learn

Describe the act–observe loop.
Show actions depending on observations.
Measure success over a multi-step task.
See an unreliable tool propagate errors.

Workshop

The character model fine-tuned for 200 steps on 30 demonstrated trajectories of three-number sums (digits 1–9, the first two summing to at least 10), loss on its own characters. Held-out: 30 sums, greedy. A faulty calculator returns a wrong result (off by 1 to 9) with the chosen probability.

The second call depends on the first

With a reliable calculator the agent solves 77% of held-out sums, making 2.00 calls per episode on average: it copies the result of the first call into the second, exactly the dependence on observations that makes this a loop rather than a pair of independent calls. Its failures are almost all copying slips. When the calculator is wrong 10% of the time, success falls to 73%; at 30% to 47%; at 50% to 27%. Every wrong result flows straight into the next call and the answer, because nothing in the agent’s training ever showed it a result worth doubting.

Real agents face unreliable observations constantly — flaky tests, stale web pages, tools that time out — and the remedies are the ones a careful person uses: check results against expectations, retry, use a second source, and say so when unsure. These behaviours do not appear by themselves; they come from training on trajectories that contain them, or from reinforcement learning in environments where checking pays (Vol. XXVI). The loop also costs context: every action and observation stays in the window, which is the subject of plate XXV.4.

The model sees its own earlier actions in the context exactly as it sees the observations; only the harness knows which text came from where — which is why loss masks (plate XXVI.5) and trust boundaries (plate XXV.8) must be kept outside the model.

Terms

Agent: A model run in a loop of actions and observations.
Observation: What the environment returns after an action.
ReAct: Interleaving written reasoning with actions in the loop.
Episode: One run of the loop from task to final answer.

An agent is a model in an act–observe loop; later actions depend on observations, so a wrong observation propagates unless the agent learned to check.

Sources

Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models”, ICLR 2023.
Nakano et al., “WebGPT: Browser-assisted question-answering with human feedback”, arXiv:2112.09332, 2021.