Vol. XX · Safety Training Plate 9 of 10safety evaluation, red-teaming, capability evaluation, propensity, rare failures, rule of three, statisticsAE

Evaluating Safety.

Zero failures in a thousand tries does not mean zero. It means probably fewer than three in a thousand.

Two kinds of safety evaluation answer different questions. Capability evaluations ask whether a model could do something dangerous if it tried or were directed to — they set the stakes and are run with the model’s full effort elicited. Propensity evaluations ask whether it would, under realistic conditions — whether it complies with harmful requests, deceives, or acts against instructions. Red-teaming searches for failures deliberately, by experts and by automated methods, and is most useful as a continuing process that feeds training (plate XX.4) rather than a test passed once.

All of these meet the same statistical fact: the failures that matter are rare, and rare events are hard to see. If a model fails on a fraction p of attempts, n independent attempts show at least one failure with probability 1 − (1 − p)ⁿ; and if n attempts show none, the most that can be said with 95% confidence is that p is below about 3/n. The workshop computes both.

Below: the chance of seeing at least one failure against the number of attempts, for the chosen failure rate and for rates ten times higher and lower; readouts give the attempts needed and the bound from a clean result.

What you will learn
Workshop
Independent attempts with a constant failure probability p. Attempts for a 95% chance of at least one observed failure: ln 0.05 / ln(1 − p). Upper bound after n clean attempts: the rule of three, 3/n, an approximation to the exact 95% bound 1 − 0.05^{1/n}.

Rare failures, finite evidence

A failure that occurs once in a thousand attempts is seen in 1 000 attempts only 63.2% of the time; seeing it with 95% confidence takes about 2,994 attempts. If 1 000 attempts show no failure, the honest conclusion is that the rate is probably below 3.00 × 10⁻³ — not that it is zero. At one in ten thousand, about 30 thousand attempts are needed. A deployment serving millions of requests will meet failures that no pre-release evaluation of practical size could have observed.

This is why safety evaluation is a process with several parts rather than a pass mark. Red-teaming concentrates attempts where failures are likely, so it finds rare failures far faster than random sampling — at the price of saying little about the rate on ordinary traffic. Capability evaluations bound what is at stake. Monitoring after deployment watches the real distribution. And claims about safety should state what was tested, how many times and how, so that a clean result is read as the bound it is.

Attempts are rarely independent: a red-teamer who finds one failure finds its neighbours, and automated search concentrates near past successes. That speeds discovery and complicates the arithmetic of rates, which assumes independent draws.

Terms

Capability evaluation
Testing what a model could do, with its full effort elicited.
Propensity evaluation
Testing what a model would do under realistic conditions.
Red-teaming
Searching deliberately for failures, by people and by automated methods.
Rule of three
With no failures in n independent trials, the 95% upper bound on the rate is about 3/n.
Rare failures need many attempts to see; zero failures in n attempts bounds the rate below about 3/n, nothing more.

Sources

  1. Hanley and Lippman-Hand, “If nothing goes wrong, is everything all right?”, JAMA 249, 1983 (the rule of three).
  2. Shevlane et al., “Model evaluation for extreme risks”, arXiv:2305.15324, 2023.
  3. Perez et al., “Red Teaming Language Models with Language Models”, EMNLP 2022.
  4. Ganguli et al., “Red Teaming Language Models to Reduce Harms”, 2022.