Vol. XX · Safety Training Plate 5 of 10honesty, calibration, confidence, reliability diagram, expected calibration error, overconfidenceLE

Honesty and Calibration.

Of the answers it is 90% sure of, are 90% right?

Honesty for a model has several parts: not asserting what it believes false, not creating false impressions, and expressing uncertainty where it has it. The last is measurable. A model is calibrated if its stated or implied confidence matches its accuracy — of all answers given with probability 0.8, about 80% are right. A reliability diagram bins answers by confidence and plots accuracy against confidence; the expected calibration error (ECE) is the average gap, weighted by the number of answers in each bin.

Pre-trained language models are often well calibrated on their next-token probabilities; the GPT-4 technical report showed post-training could degrade this. The workshop measures calibration directly on the post-training world: for each of the hundred sums, the model’s greedy answer and the probability it assigned to it, for the fine-tuned model of plate XVII.1 and for the same model after the RL of plate XIX.5.

Below: the reliability diagram — accuracy against confidence in five bins — for the chosen model, with perfect calibration as the diagonal; readouts give the expected calibration error.

What you will learn
Workshop
All 100 sums in chat form; confidence is the product of the probabilities of the greedy answer’s tokens, end mark included; correct if the answer equals the sum. Five equal-width confidence bins; empty bins omitted. RL model: GRPO, 40 steps (plate XIX.5).

Saying how sure it is

The fine-tuned model gives its answers with a mean confidence of 86% but is right on 70%: its confidence exceeds its accuracy by 0.16 on average, with an expected calibration error of 0.156. Most of its answers sit in the top confidence bin, and fewer of those are right than their confidence claims. After reinforcement learning against the verifier, accuracy rises to 85%, mean confidence to 90%, and the calibration error falls to 0.052.

That RL here improved calibration is worth noticing because the opposite has also been reported: a reward that pays only for being right pushes probability towards answers the model can get right and, on a verifiable task, can make confidence more informative; a reward from human preferences can instead reward sounding sure (plate XVIII.10). Calibration is the part of honesty that can be checked by counting; the parts that cannot — whether a model’s stated reasons are its real ones, whether it is concealing something — are among the hardest open problems, taken up in plate XXX.8.

Calibration can be measured on the probabilities a model assigns or on the confidence it states in words; the two can disagree. Models can be trained to state calibrated verbal confidence, which matters because users read words, not logits.

Terms

Calibration
Agreement between confidence and accuracy.
Reliability diagram
Accuracy plotted against confidence, in bins.
ECE
Expected calibration error: the weighted mean gap between confidence and accuracy.
Overconfidence
Confidence systematically above accuracy.
Calibration makes honesty about uncertainty measurable: of answers given with confidence p, a share p should be right.

Sources

  1. Guo, Pleiss, Sun and Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017.
  2. Kadavath et al., “Language Models (Mostly) Know What They Know”, arXiv:2207.05221, 2022.
  3. OpenAI, “GPT-4 Technical Report”, arXiv:2303.08774, 2023 (calibration before and after post-training).
  4. Evans et al., “Truthful AI”, arXiv:2110.06674, 2021.