Drills: Evaluation.
Five questions at a time from the whole volume, until every score you read comes with a question attached.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that a score is a measurement: of a model with a prompt, on a sample of items, under a scoring rule, with an error bar and a history — and each of those can be the larger part of what the number says.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Read benchmarks as measurements.
- Put error bars on scores.
- Estimate pass@k correctly.
- Spot judge bias, leakage and saturation.
What to carry forward
A benchmark measures a skill in a format under a rule (plate XXI.1); its score has an error bar of about 2√(p(1 − p)/n) (plate XXI.2) and moves with the prompt (plate XXI.3); pass@k needs the unbiased estimator (plate XXI.4); LLM judges have position and length biases (plate XXI.5).
Arena ratings are preference measured with Bradley–Terry (plate XXI.6); leaked items measure memory (plate XXI.7); time horizons measure long tasks (plate XXI.8); and tests inform only near the abilities they are pitched at (plate XXI.9). Vol. XXII turns from measuring outputs to producing them: decoding.
Terms
- Construct validity
- Measuring what the name claims.
- Standard error
- √(p(1 − p)/n).
- pass@k
- 1 − C(n − c, k)/C(n, k).
- Contamination
- Benchmark items in training data.