Vol. XXI · Evaluation Plate 9 of 10evaluation design, item response theory, difficulty, discrimination, test information, validity, gradingEL

Designing an Evaluation.

Questions pitched at the right difficulty tell models apart. Questions too easy or too hard do not.

Psychometrics, the science of designing tests for people, has a model for what makes a question informative. In item response theory, a test-taker of ability θ answers an item of difficulty b correctly with probability σ(a(θ − b)), where a is the item’s discrimination. An item tells test-takers apart most where its success curve is steepest — at abilities near its difficulty — and the test’s information at an ability is the sum of its items’ contributions there. Applied to language models, IRT can choose small subsets of a benchmark that rank models as well as the whole, and explains why benchmarks saturate.

Design involves more than difficulty: questions must test the intended capability (validity), have unambiguous correct answers or rubrics (reliable grading), be free from contamination (plate XXI.7), and come with a plan for analysis (plates XXI.2–XXI.3). The workshop builds a 40-item test with difficulties spread around a chosen centre and shows its scores and information across abilities.

Below: left, the test score of a model of each ability for the chosen design. Right, the test’s information across abilities — where it can tell models apart.

What you will learn
Workshop
40 items with difficulties spaced evenly over [centre − spread/2, centre + spread/2], all with the chosen discrimination; abilities from −4 to 4. Score: the expected share correct. Information: Fisher information of the test.

Pitch the questions where the models are

With difficulties spread over four units around zero, the test is most informative near ability 0.00, and two strong models at abilities 1.5 and 2.5 differ by 12.6% on it. Make every question easy — difficulties packed near −2 — and both strong models score near the top: the gap shrinks to 0.4%, the benchmark has saturated for them. Spread the difficulties wider and the test measures across a broader range at the cost of fewer items near any one ability: a gap of 13.4%.

This is why new benchmarks are built from questions the current best models fail, and why a benchmark that once ranked models well stops doing so as they improve. IRT also allows smaller tests: tinyBenchmarks showed that about a hundred well-chosen items can estimate a model’s score on a benchmark of thousands. But difficulty is only one requirement. A question that is hard because it is ambiguous measures nothing; an answer key with errors caps every score; and a test of the wrong capability is precisely measured and beside the point.

Discrimination matters as much as difficulty: an item that strong and weak models get right with similar probability — because it is noisy, ambiguous or mislabelled — adds length to a test without adding information.

Terms

Item response theory
A model of the probability of answering an item correctly from ability and item parameters.
Difficulty
The ability at which an item is answered correctly half the time.
Discrimination
How sharply an item separates abilities around its difficulty.
Test information
How precisely a test measures ability at each level.
A test measures well only near the abilities its questions are pitched at; easy items saturate, impossible ones measure nothing.

Sources

  1. Lord, Applications of Item Response Theory to Practical Testing Problems, 1980.
  2. Polo et al., “tinyBenchmarks: evaluating LLMs with fewer examples”, ICML 2024.
  3. Bowman and Dahl, “What Will it Take to Fix Benchmarking in Natural Language Understanding?”, NAACL 2021.