Vol. XXI · Evaluation Plate 3 of 10prompt sensitivity, format variance, few-shot choice, robustness of evaluation, prompt engineering, varianceEL

One Prompt, Many Scores.

Ask the same question six ways and get six scores. Which one is the model’s?

Language models are sensitive to the exact wording of a prompt: spacing, separators, the phrasing of the question, the choice and order of few-shot examples. Sclar and colleagues (2024) measured accuracy differences of tens of points across semantically equivalent prompt formats for the same model, and found that format choice could reverse the ranking of two models. A benchmark score is therefore a measurement of a model with a particular prompt, and an honest report states the prompt or averages over several.

The workshop takes the fine-tuned model of plate XVII.1 and asks all 100 sums in six formats: the one it was trained on, with spaces around “+”, with “=?”, as “what is …?”, without the question mark, and as “add a and b”. It also asks them in the trained format with two random filler words, eight times with different fillers.

Below: left, accuracy on the 100 sums for each of six prompt formats. Right, accuracy for eight draws of random filler words around the same questions.

What you will learn
Workshop
The fine-tuned model, greedy answers, all 100 sums. Formats: “U:a+b?|A:” (trained), “U:a + b?|A:”, “U:a+b=?|A:”, “U:what is a+b?|A:”, “U:a+b|A:”, “U:add a and b|A:”. Filler: two filler words drawn at random, in the trained format, eight seeds.

The format is part of the measurement

In the format it was fine-tuned on, the model answers 70% of the sums; in the worst of the five rewordings, 53%. The six formats span 17%, against a sampling error of about 4.9% for any one of the scores — the format effect is several times the statistical noise. Even within the trained format, eight random choices of two filler words give accuracies spanning 9%.

Larger and better-trained models are less sensitive than this tiny one, but not insensitive, and their benchmark differences are often only a few points. Good practice therefore treats the prompt as part of the experiment: publish it, use the format the model was designed for or several formats, and report the spread. Evaluation suites such as HELM standardise prompts across models for exactly this reason — at the cost of a format that may suit some models better than others.

Few-shot examples add their own variance: which examples are chosen and in what order can move accuracy by many points, partly because models over-weight the labels of the most recent examples.

Terms

Prompt sensitivity
Changes in output or score from changes in wording that should not matter.
Format
The surface layout of a prompt: separators, phrasing, spacing.
Few-shot examples
Worked examples included in a prompt.
Spread
The range of scores across equivalent prompts.
A score is a measurement of a model with a prompt; equivalent wordings can move it by more than its statistical error.

Sources

  1. Sclar, Choi, Tsvetkov and Suhr, “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design”, ICLR 2024.
  2. Zhao et al., “Calibrate Before Use: Improving Few-Shot Performance of Language Models”, ICML 2021.
  3. Liang et al., “Holistic Evaluation of Language Models”, TMLR 2023 (HELM).