Vol. XX · Safety Training Plate 6 of 10hallucination, confabulation, abstention, confidence threshold, coverage, scoring rules, evaluation incentivesLE

Hallucination.

A confident wrong answer and a confident right one look the same — unless the model knows the difference.

Hallucination — fluent, specific, false output — follows from how models are trained and graded. Pre-training rewards continuing text plausibly, not stopping when unsure. And most benchmarks score an answer 1 if right and 0 otherwise, so a guess has positive expected score while “I don’t know” scores zero: as Kalai and colleagues (2025) argued, evaluations themselves reward guessing. A model that has learned to be helpful will answer, and when its knowledge runs out the answer is invented.

The remedy available to every model is to use its own uncertainty: answer when confident, abstain otherwise. It works only as well as the model’s confidence separates its right answers from its wrong ones. The workshop uses the post-training world’s fine-tuned model: for each of the 100 sums, its answer and confidence; it abstains below a threshold and is scored with a chosen penalty for wrong answers.

Below: left, how many answers are right and wrong at each confidence level. Right, coverage, accuracy of the answers given, and the score under the chosen penalty, against the abstention threshold.

What you will learn
Workshop
The fine-tuned model of plate XVII.1 on all 100 sums: greedy answer, confidence (product of its token probabilities) and correctness. Abstaining scores 0, a right answer 1 and a wrong answer −λ, averaged over the 100. Threshold sweep from 0 to 1.

Abstain when it pays

Answering every question, the model is wrong on 30 of 100. If a wrong answer costs as much as a right one earns (λ = 1), always answering scores 0.400; abstaining below confidence 0.5 answers 97% of the questions, gets 72% of those right, and scores 0.430. When wrong answers cost nothing (λ = 0), abstaining can only lose — the incentive most benchmarks create. When they cost three times as much, abstaining below 0.75 scores 0.060, against −0.200 for answering everything.

Two things limit the remedy. The model’s confidence must separate right from wrong, and the bars show it only partly does: some wrong answers come with high confidence, which no threshold removes. And the incentive must exist: if evaluations and users reward a confident answer over an honest “I am not sure”, training will produce confident answers. Retrieval of sources (plate XXVI.1), training on abstention, and scoring rules that penalise errors more than abstentions all push the same way; none makes a model know what it does not.

Hallucination is distinct from lying: a model that hallucinates does not represent the claim as false and assert it anyway; it produces a plausible continuation where it has no reliable knowledge. The distinction matters for how each is detected.

Terms

Hallucination
Fluent, specific output that is false or unsupported.
Abstention
Declining to answer, or saying it does not know.
Coverage
The share of questions answered.
Scoring rule
How right, wrong and abstaining answers are scored.
Models hallucinate because training and grading reward a plausible answer over “I don’t know”; abstaining on low confidence helps as far as confidence separates right from wrong.

Sources

  1. Kalai, Nachum, Vempala and Zhang, “Why Language Models Hallucinate”, arXiv:2509.04664, 2025.
  2. Ji et al., “Survey of Hallucination in Natural Language Generation”, ACM Computing Surveys 55, 2023.
  3. Kadavath et al., “Language Models (Mostly) Know What They Know”, 2022.