Vol. XXI · Evaluation Plate 2 of 10error bars, confidence intervals, binomial, standard error, sample size, statistical significance, paired comparisonEL

Error Bars.

Fifty-three per cent on thirty questions means somewhere between thirty-five and seventy.

A benchmark is a sample from the questions it could have contained. A model with true accuracy p scores p̂ on n questions, and p̂ varies from sample to sample with standard error √(p(1 − p)/n): about 9 points at n = 30, 2 points at n = 500, half a point at n = 10 000. The 95% interval is roughly p̂ ± 2 standard errors. Miller (2024) argued that most published language-model comparisons omit this, and that many reported differences are within noise.

Comparing two models is easier than measuring one if they answer the same questions: a paired comparison counts only the questions where they disagree, removing the variation from question difficulty that both share. The workshop computes intervals and the number of questions needed to tell two models apart, starting from this course’s own small held-out sets of 30.

Below: left, the 95% interval of a score against the number of questions. Right, the number of questions needed to separate two models with the chosen accuracies, unpaired and paired.

What you will learn

Compute the standard error of a benchmark score.
Draw a 95% interval.
Compute the questions needed to separate two models.
Explain the advantage of paired comparisons.

Workshop

Normal approximation to the binomial. Questions needed for a difference p₁ − p₂ at 5% significance and 80% power (z = 1.96 + 0.84) when the two models answer independent question sets; paired, the variance is reduced by the questions on which both models agree, assumed here to be the chosen share.

Thirty questions say little

This course has quoted held-out accuracies on 30 questions, and they deserve their error bars: 53% on 30 questions has a standard error of 9.1% and a 95% interval of 35% to 71%. Separating a 53% model from a 58% one would take about 1,545 independent questions, or 464 if both answer the same questions and agree on 70% of them. Separating 80% from 82% takes 6,029 unpaired. The differences between runs in Vols. XVII–XX that were a few points apart are within noise; the large ones — 17% against 70%, 0% against 87% — are not.

Three habits follow. Report intervals, not bare scores. Compare models on the same questions and analyse the disagreements, which is far more powerful than comparing two separate averages. And remember that the interval covers only sampling of questions: it says nothing about variation from prompt format (plate XXI.3), sampling temperature or the random seed of training, each of which can move a score by more than the error bar suggests.

Questions in a benchmark are often clustered — many items from one passage or one problem template — which makes the effective sample size smaller than the item count and the true error bars wider than the binomial formula gives.

Terms

Standard error: The typical sampling variation of an estimate.
Confidence interval: A range that contains the true value with stated probability under repeated sampling.
Paired comparison: Comparing two models on the same items, using their disagreements.
Statistical power: The chance of detecting a real difference of a given size.

A score on n questions is uncertain by about 2√(p(1 − p)/n); small differences need thousands of questions, or paired comparisons.

Sources

Miller, “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations”, arXiv:2411.00640, 2024.
Card et al., “With Little Power Comes Great Responsibility”, EMNLP 2020.