Vol. XXII · Decoding Plate 6 of 8calibration, reliability diagram, expected calibration error, confidence, overconfidence, post-trainingAE

Calibration.

Of the tokens the model gives 30 per cent, are 30 per cent right?

Decoding rules use the model’s probabilities as if they meant something, and so does anyone who reads a confidence from a model. They mean something only if the model is calibrated: among all predictions made with probability around p, a share p turns out correct. The check is a reliability diagram — bin the predictions by confidence and plot each bin’s accuracy against its mean confidence — summarised by the expected calibration error (ECE), the average gap weighted by how many predictions fall in each bin.

Cross-entropy is a proper scoring rule: it is minimised only by reporting true probabilities, so a model trained to predict the next token is pushed towards calibration on that task. The course model micro (1.8 million parameters, four layers, trained on this course’s own plates in English and Chinese; plate I.9) runs in the page. The workshop measures micro’s top-token confidence against whether the top token was the actual next token, on course plates held out from its training.

Below: the reliability diagram of micro’s next-token predictions on held-out text; readouts give the expected calibration error, the mean confidence and the accuracy.

What you will learn
Workshop
micro on held-out course text — the first 2 400 characters of the English and 800 of the Chinese held-out plates (every fiftieth plate, never trained on) — read in windows of the model’s context. Confidence: probability of the most probable next token. Correct: that token was the actual next token. Ten equal-width bins; empty bins are omitted. Temperature rescales the logits before the probabilities are read.

Pretrained models know how sure to be

On 1,167 next-token predictions micro is right 26% of the time with a mean confidence of 26%; the expected calibration error at temperature 1 is 0.025. The points lie near the diagonal: a model with 1.8 million parameters is wrong most of the time and knows it. Sharpen its probabilities with temperature 0.5 and the error grows to 0.334 — overconfidence — and flatten them with 2 and it grows to 0.228 the other way.

This is why temperature 1 means “the model’s own distribution”, and why sampling at 1 writes typical text. Post-training changes the objective: preference optimisation and RL reward the answer people prefer, not the probability that it is right, and published measurements show the effect — the GPT-4 report shows a pretrained model close to calibrated on a multiple-choice benchmark and the post-trained one markedly overconfident. A model’s verbal confidence (“I’m fairly sure”) is a separate output again, and must be checked separately.

Fitting a single temperature on held-out data — temperature scaling — is the standard fix for an overconfident classifier and leaves its accuracy unchanged, because it does not change the ranking (plate XXII.2).

Terms

Calibration
Agreement between predicted probabilities and observed frequencies.
Reliability diagram
Accuracy against confidence, bin by bin.
Expected calibration error
The weighted average gap between confidence and accuracy.
Proper scoring rule
A loss minimised only by reporting true probabilities, such as cross-entropy.
Next-token training pushes a model towards calibration, because cross-entropy is a proper scoring rule; post-training optimises something else and can undo it.

Sources

  1. Guo, Pleiss, Sun and Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017 (ECE, temperature scaling).
  2. Kadavath et al., “Language Models (Mostly) Know What They Know”, arXiv:2207.05221, 2022.
  3. OpenAI, “GPT-4 Technical Report”, arXiv:2303.08774, 2023 (calibration before and after post-training, figure 8).