Vol. XXV · Tools and Agents Plate 5 of 9compounding errors, reliability, long tasks, error recovery, pass^k, time horizon, agentsCE

Errors Compound.

Ninety-nine per cent per step is thirty-seven per cent per hundred steps.

A long task is a chain of steps, and a chain is as strong as the product of its links. If each step succeeds independently with probability p and one failure ruins the task, n steps succeed with probability pⁿ — a curve that falls off a cliff as tasks lengthen. This is the mechanism behind the time horizons of plate XXI.8: success falls with task length because long tasks contain more chances to fail. Agents that notice and repair some of their errors break the product: if a fraction r of errors is caught and fixed, the effective per-step success is p + (1 − p)r.

Reliability also has a second meaning. τ-bench measures pass^k, the chance an agent succeeds on all of k independent tries of the same task — what a user who returns to it every day experiences — and finds it far below the single-try rate. The workshop computes both from the chosen per-step success and recovery rate.

Below: left, task success against the number of steps for the chosen per-step success, with and without recovery; right, pass^k for a task solved with the resulting probability.

What you will learn
Workshop
Independent steps; an uncaught error fails the task; a caught error is repaired at no cost. pass^k assumes independent tries of the same task at the given success probability — real tries of one task are correlated, so measured pass^k is usually higher than this for easy tasks and lower for hard ones.

Reliability sets the horizon

At 99% per step a 100-step task succeeds 37% of the time, and half of all tasks fail by step 69; at 99.9% a 1 000-step task succeeds 37% of the time. Improving per-step reliability by a factor of ten in error rate multiplies the length of task an agent can finish by ten. Catching and fixing half of its errors does the same as halving the error rate: at 99% per step the 50% point moves from 69 to 138 steps.

This is why a model that is brilliant at single steps can still be a weak agent, and why agent training focuses on noticing failure — running the tests, re-reading the output, checking an answer by another route — as much as on getting steps right the first time. τ-bench found its best agent succeeded on under 50% of tasks once and on under 25% of retail tasks in all of eight tries: consistency is a capability of its own, and it is what users experience.

Real steps are not independent: one misunderstanding early can doom every later step, while easy stretches carry no risk. The pⁿ model is a lower bound on difficulty for hard tasks and an upper one for easy ones — a guide to scaling, not a forecast.

Terms

Compounding error
Task failure probability growing with the number of steps.
Recovery
Noticing and repairing an error before it ruins the task.
pass^k
The chance of succeeding on all of k tries of a task.
Reliability
Consistency of success across steps and tries.
Task success is roughly per-step success to the power of the steps; recovering from errors changes the exponent’s base, which is why reliability limits long tasks.

Sources

  1. Kwa, West et al. (METR), “Measuring AI Ability to Complete Long Tasks”, arXiv:2503.14499, 2025.
  2. Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, arXiv:2406.12045, 2024.