Vol. XIX · Reinforcement Learning for Reasoning Plate 10 of 12test-time compute, self-consistency, majority voting, best-of-n, verifier, search, reasoning models, inference scalingLE

Spending Compute at Test Time.

More samples, more votes, longer thoughts: accuracy bought at inference instead of training.

Training is not the only place to spend compute. At inference a model can sample n answers and return the most common one — self-consistency (Wang et al., 2022) — or return one a verifier accepts, which with a perfect verifier succeeds whenever any sample is right (pass@n). It can search over partial solutions with a process reward model, or simply write a longer chain of thought before answering. OpenAI’s o1 (2024) and DeepSeek-R1 (2025) made the last one the headline: models trained by RL to think at length, whose accuracy rises with the tokens they are allowed to spend.

Snell and colleagues (2024) found that for a fixed budget, choosing the right strategy per problem could match a model many times larger. The workshop spends test-time compute on the RL world’s fine-tuned model: for each held-out question it draws up to 32 samples at a chosen temperature, and compares majority voting with a perfect verifier as the number of samples grows.

Below: held-out accuracy against the number of samples, for majority voting and for best-of-n with a perfect verifier, beside the single greedy answer; right, the answers drawn for one question.

What you will learn
Workshop
The fine-tuned model of plate XVII.1, 32 samples per held-out question at the chosen temperature. Majority vote over the first n samples (ties broken by first occurrence); best-of-n with a perfect verifier is the unbiased pass@n from the 32 samples. Greedy: the single most probable answer.

Votes saturate; verifiers do not

The greedy answer is right on 53% of the held-out questions. Majority voting over 32 samples at temperature 1 reaches 57%: where the model’s most common answer is wrong, more votes only confirm it. Best-of-32 with a perfect verifier reaches 73%, because it needs only one right sample among the 32. At temperature 0.5 the samples agree more and voting adds little (57%); at 1.5 they spread, and the verifier finds more (80%).

The general shape holds at scale: voting improves accuracy and then plateaus at the model’s most common answer, while selection keeps improving as long as the selector is reliable — which makes verifiers, and the problems that have them, the place where test-time compute pays most. Reasoning models fold much of this inside one long generation: trained by RL, they explore, check and revise within a single chain of thought, and their accuracy grows with the length of that chain. The cost is latency and tokens per answer, which serving systems must budget (Vol. XXIII).

Majority voting needs answers that can be compared for equality — numbers, multiple choices, short strings. For open-ended text, the equivalent is a judge or reward model choosing among samples, with that judge’s biases (plate XVIII.6).

Terms

Test-time compute
Computation spent at inference to improve an answer.
Self-consistency
Sampling several reasoning paths and taking the most common answer.
Best-of-n
Sampling n answers and keeping the one a verifier or reward model prefers.
Reasoning model
A model trained to think at length before answering.
Test-time compute buys accuracy at inference: voting saturates at the model’s favourite answer; a reliable verifier keeps paying.

Sources

  1. Wang et al., “Self-Consistency Improves Chain of Thought Reasoning in Language Models”, ICLR 2023.
  2. Snell, Lee, Xu and Kumar, “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters”, arXiv:2408.03314, 2024.
  3. OpenAI, “Learning to Reason with LLMs” (o1), September 2024; DeepSeek-AI, DeepSeek-R1, 2025.