Vol. XX · Safety Training Plate 3 of 10refusal, over-refusal, under-refusal, false refusal, trade-off curve, judgement, XSTestAE

Refusal and Over-Refusal.

Refusing everything is safe and useless. The real measure is how few legitimate requests are turned away.

A model can fail in two directions. Under-refusal is helping with a request that should have been declined. Over-refusal is declining a legitimate request because it superficially resembles a harmful one — a question about a medication dose, a historical atrocity, a security concept — which makes the assistant useless exactly where people need careful help. Test sets such as XSTest (Röttger et al., 2024) measure the second with prompts that are safe but sound unsafe.

The two rates move against each other as the decision threshold moves, tracing a curve; where a model sits on it is a policy choice (plate XX.2), but the curve itself is set by how well the model tells harmful from legitimate requests. The workshop varies that judgement — the noise with which the stylised model perceives harm — and shows the whole curve shift.

Below: the trade-off curve between over-refusal and under-refusal as the threshold varies, for the chosen quality of judgement and, for comparison, a much better one; the chosen threshold is marked.

What you will learn

Define over-refusal and under-refusal.
Draw their trade-off curve.
Show that judgement, not threshold, moves the curve.
Name how over-refusal is measured.

Workshop

The stylised refusal model of plate XX.2 (400 requests; 75% benign, 15% sensitive but legitimate, 10% harmful). Each curve sweeps the threshold from 0 to 1 at a fixed perception noise σ; the dot marks the chosen threshold. Comparison curve: σ = 0.03.

Move the curve, not along it

With judgement noise 0.12 and threshold 0.5, the model turns away 9.5% of legitimate requests and helps 4.9% of harmful ones; the best any threshold can do on this curve is a combined 12.8%. With much better judgement the curve moves towards the corner, and the best combined rate is 4.2%. With poor judgement (0.3) the best is 41.8%; with good judgement (0.04), 4.7%.

This is why “make it refuse less” and “make it safer” are not opposed goals once the model is good enough: both are served by better discrimination. It also explains a common pattern in safety training. Training that adds refusal data without teaching distinctions moves the model along its curve — fewer harmful requests helped, more legitimate ones refused — and over-refusal benchmarks catch it. Training that teaches the distinctions, with examples of legitimate requests that resemble harmful ones and of careful partial answers, moves the curve itself.

Refusal is not the only safe response: specifications increasingly prefer a safe completion — helping with the legitimate part of a request, giving high-level information, or redirecting — over a flat “I can’t help with that”.

Terms

Over-refusal: Declining a legitimate request because it resembles a harmful one.
Under-refusal: Helping with a request that should have been declined.
Trade-off curve: The pairs of error rates reachable by varying the threshold.
Safe completion: A response that helps with what is legitimate in a request while withholding what is not.

Over- and under-refusal trade along a curve set by the model’s judgement; thresholds move along it, better discrimination moves the curve.

Sources

Röttger et al., “XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models”, NAACL 2024.
Cui et al., “OR-Bench: An Over-Refusal Benchmark for Large Language Models”, 2024.
OpenAI, “Model Spec”, 2024 (prefer helpful, safe completions).