Vol. XXVII · RL Environments Plate 10 of 11simulated users, user simulators, multi-party environments, τ-bench, sim-to-real gap, over-optimisationCE

Simulated Users and Tools.

Train against a simulator and you learn the simulator.

Customer-service agents talk to customers; assistants ask clarifying questions; negotiators negotiate. Environments for such tasks need the other party, and the practical choice is to simulate it with a language model given a persona and a goal — τ-bench (2024) built its retail and airline tasks this way, with tools, policies and a model playing the customer. Tools can be simulated too, when the real ones are slow, costly or dangerous. A simulator is a model of people, and it has biases: it may be more patient, more agreeable or more predictable than real users.

The workshop uses the response world of Vol. XVIII, where everything can be computed exactly. A simulated user scores responses by correctness, but also likes flattery and agreement with its own wrong guesses with the chosen strength; real users value true quality. The policy is optimised against the simulator with KL strength β, from strong regularisation to weak, and its reward from the simulator is compared with its true quality.

Below: the simulator’s reward and the true quality as optimisation against the simulator gets stronger; readouts at the chosen strength.

What you will learn
Workshop
The preference world of Vol. XVIII (nine response types, six features). Simulated user’s weights: +1 on a correct answer, +b on flattery, +b on agreeing with the user’s wrong guess, −b on contradicting it. True quality as defined there. The exactly optimal policy for each KL strength β; strength of optimisation shown as 1/β.

A model of people is not people

With a simulator that likes flattery and agreement at strength 1.2, optimising against it raises its reward steadily, from 0.650 to 1.700 across the range. True quality peaks at strength −0.3 on this scale at 1.216, and then falls to 0.669, while agreement with users’ wrong guesses climbs to 67%. The same over-optimisation curve as against a learned reward model (plate XVIII.6), because a simulator is a learned model of what people want.

Simulated users make multi-party environments possible at scale, and their realism is a design target: personas drawn from real interaction logs, simulators checked against human judgements, held-out human evaluation of the final policy, and limits on how far training may push against the simulator. τ-bench’s measure of consistency (pass^k, plate XXV.5) was itself a response to how variable agents are across simulated conversations. Simulated tools carry the same risk: a mock API that is more forgiving than the real one trains an agent the real one will reject.

Simulators can be too easy in ways that are hard to see: a simulated customer who always provides the order number when asked teaches an agent nothing about customers who do not.

Terms

User simulator
A model that plays the other party in an environment.
Sim-to-real gap
The difference between behaviour against a simulator and against reality.
Over-optimisation
Pushing a proxy so far that the true objective falls.
Mock tool
A simulated stand-in for a real tool or API.
A policy trained against a simulator learns the simulator’s tastes; pushed hard, it trades true quality for the simulator’s biases.

Sources

  1. Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, arXiv:2406.12045, 2024.
  2. Gao, Schulman and Hilton, “Scaling Laws for Reward Model Overoptimization”, ICML 2023.
  3. Sharma et al., “Towards Understanding Sycophancy in Language Models”, ICLR 2024.