Vol. XXI · Evaluation Plate 1 of 10benchmarks, evaluation, construct validity, format, ranking, capability, measurementEL

Benchmarks.

A benchmark measures a capability in one format, with one rule. Change either and the ranking changes.

Benchmarks made progress in language models measurable: MMLU for knowledge across subjects, GSM8K and MATH for mathematics, HumanEval and SWE-bench for code, and hundreds more. Each is a set of questions, a format in which they are posed, and a rule for scoring the answers — and each measures the intersection of all three. A model can know the answer and fail the format; a model can learn the format of a benchmark without the skill behind it. Construct validity is the question of whether the score measures the capability its name claims.

The post-training world has four models — the pre-trained base, the fine-tuned chat model, the chat model after RL, and the chain-of-thought model — and four small benchmarks built from questions none of them trained on: sums in chat form, sums in worksheet form, chat sums wrapped in filler words, and three-number sums. The workshop scores every model on every benchmark.

Below: the accuracy of each of the four models on each of the four benchmarks; the chosen benchmark’s ranking is listed in the readouts.

What you will learn
Workshop
Held-out items only. Chat sums: the 30 held-out sums as “U:a+b?|A:”. Worksheet sums: the same sums after “1+1=2. ”. Chat with filler: chat sums with four filler words. Three-number sums: 30 held-out triples in chat form, scored on the final number. Greedy answers.

Every model is best at something

On chat sums the RL-trained model leads (73%, against 53% for the fine-tuned model it came from). On worksheet sums the base model is perfect (100%) — it was pre-trained on exactly that format — while it scores zero in chat form, because it never learned to answer. On three-number sums the chain-of-thought model scores 70% and the others almost nothing (10% for the chat model), yet it scores 0% on plain chat sums, which it was never fine-tuned to answer in that format. With filler words the RL model scores 50%, against 73% without them.

None of these models is “best”; each is best on the benchmark that matches its training. Real leaderboards show the same structure less starkly: a model tuned for a benchmark family leads it, rankings reorder across families, and aggregate scores hide the reordering. Reading a benchmark well means asking what format it poses questions in, what it rewards, what its items are drawn from, and whether the capability named in its title is the one being measured — the subject of the rest of this volume.

Format effects are largest for models that were not trained on a format at all; fine-tuned assistants are more robust to rewording than base models, but the plates that follow show they are not immune (plate XXI.3).

Terms

Benchmark
A fixed set of questions with a format and a scoring rule.
Construct validity
Whether a score measures the capability it claims to.
Format
How questions are posed and answers are expected.
Aggregate score
A single number averaged over many benchmarks.
A benchmark measures a skill in one format under one rule; change any of the three and the ranking can change.

Sources

  1. Hendrycks et al., “Measuring Massive Multitask Language Understanding”, ICLR 2021 (MMLU).
  2. Cobbe et al., “Training Verifiers to Solve Math Word Problems”, 2021 (GSM8K).
  3. Raji et al., “AI and the Everything in the Whole Wide World Benchmark”, NeurIPS Datasets and Benchmarks 2021.