Vol. XXI · Evaluation Plate 8 of 10time horizon, METR, long tasks, agents, success rate, logistic, doubling time, real-world evaluationEL

Measuring Real Tasks.

Not “can it answer?” but “how long a task can it finish?” — and how fast that is growing.

Benchmarks of short questions say little about whether a model can complete a long piece of work. METR (Kwa, West and colleagues, 2025) built a suite of 170 software and research tasks, timed skilled humans on each — from a few seconds to about eight hours — and ran frontier models on them as agents. For each model, success fell with the human time a task took, following a logistic curve in the logarithm of task length; the task length at which success is 50% is the model’s time horizon. Across models from 2019 to 2025 that horizon doubled roughly every seven months.

The workshop draws the logistic success curve for a chosen time horizon and slope, and projects the horizon forward at a chosen doubling time. The projection is an extrapolation of a trend on one family of tasks, with all the uncertainty that implies.

Below: left, the probability of completing a task against how long it takes a skilled human, for the chosen time horizon. Right, the horizon projected forward from 2025 at the chosen doubling time.

What you will learn
Workshop
Logistic success curve in log₂ of task length t with horizon h and slope β (per doubling of task length). Projection: h₀ at the start of 2025, doubling every T_d months. METR’s published horizons: about 50 minutes for Claude 3.7 Sonnet (first version of the paper) and about 110 minutes for o3 (later version), with a 2019–2025 doubling time of about seven months.

How long a task, and how fast it grows

With a 50-minute horizon — METR’s first estimate for Claude 3.7 Sonnet — and a slope of 0.8 per doubling, a one-minute task succeeds 99% of the time and an eight-hour task 7%. A 110-minute horizon, METR’s later estimate for o3, gives the eight-hour task 15%. Extrapolating a seven-month doubling, the horizon is 1,767 minutes three years on, and the trend reaches a 40-hour working week around 2028; at a four-month doubling, which METR noted recent models might be following, around 2027.

The measurement is more trustworthy than the extrapolation. Success falls with length because long tasks contain more steps that can go wrong and fewer chances to recover (plate XXV.5); the horizon summarises how reliably a model avoids and repairs errors. The tasks are software-flavoured, well specified and automatically scored, which real work often is not, and a trend fitted to a few years of models can bend. Like the scaling laws of Vol. X, it is a regularity worth tracking and a forecast worth doubting.

A logistic in log task length implies that each doubling of task length lowers the odds of success by the same factor — e^{β} — a constant “hazard” per doubling that resembles errors arriving at a steady rate.

Terms

Time horizon
The human task length at which a model succeeds half the time.
Logistic success curve
Success probability falling smoothly with log task length.
Doubling time
How long a quantity takes to double on an exponential trend.
Agentic evaluation
Running a model as an agent on multi-step tasks.
The 50% time horizon — the task length a model finishes half the time — doubled about every seven months from 2019 to 2025; trust the measurement more than the extrapolation.

Sources

  1. Kwa, West et al. (METR), “Measuring AI Ability to Complete Long Tasks”, arXiv:2503.14499, 2025.
  2. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024.
  3. Ord, “Is there a half-life for the success rates of AI agents?”, 2025 (constant hazard reading of the curve).