Vol. XXX · The Research Frontier Plate 1 of 11reasoning models, test-time compute, chain of thought, RL with verifiable rewards, DeepSeek-R1, o1EL

Reasoning Models.

Spend computation at answer time, learned by reinforcement on checkable problems.

This volume describes the field as of October 2026. Claims are marked established (replicated or long-standing), reported (from a single lab or report, not independently reproduced) or contested (actively disputed). OpenAI’s o1 (September 2024) and DeepSeek-R1 (January 2025) made the pattern public: a model trained by reinforcement learning against verifiable rewards (Vol. XIX) learns to produce long reasoning before its answer, and accuracy on hard problems rises with how long it thinks. DeepSeek reported that R1 scored 79.8% on AIME 2024 at one sample, and that R1-Zero — RL directly from the base model with no supervised reasoning data — reached 71.0%, or 86.7% with majority voting (reported, with open weights that others have since evaluated).

Two kinds of test-time scaling are now standard and established: sampling many answers and voting or verifying (plate XIX.10), and training the model to think longer in a single answer. The workshop shows the first on the post-training world: accuracy of majority voting and pass@k as the number of samples grows, for the RL-trained model of Vol. XIX.

Below: majority-vote accuracy and pass@k against the number of samples, for the course’s RL-trained adder; readouts give the gains from sampling.

What you will learn
Workshop
The post-training world’s model after GRPO with an exact-answer reward (Vol. XIX): 32 samples at temperature 1 on each of 30 held-out sums. Vote: the most common of the first k answers. pass@k: the unbiased estimator from the 32 samples.

What changed, and what is still argued

On the course’s small adder, majority voting over 32 samples gives 70% against 70% for one sample — RL has already concentrated the model on its likeliest answer, so a vote adds little — while pass@32 is 97%: the right answer is usually among the samples, and a verifier that could pick it out would buy far more than voting. That is the established half of the reasoning-model story at every scale — more test-time computation, in samples or in thinking length, improves results on problems whose answers can be checked, and RL with verifiable rewards trains models to use it.

The contested half is what RL adds. One line of work found that RL-trained models rarely solve problems their base model could not solve at large k — RL sharpens existing abilities more than it creates new ones (plate XIX.9) — while others report genuine new capability with longer training and broader tasks. Also open: how far gains on maths and code transfer to tasks without verifiers, and how faithful the long reasoning is to the computation behind the answer (plate XXVIII.8).

The AIME results quoted are from the developers’ reports; independent evaluations of open-weight models broadly reproduced them, which is why they are treated here as well supported rather than merely claimed.

Terms

Reasoning model
A model trained to produce long reasoning before answering, usually by RL.
Test-time compute
Computation spent when answering: longer reasoning or more samples.
Verifiable reward
A reward computed by checking an answer, as in maths or code.
Majority vote
Choosing the most common answer among samples.
Reasoning models turn test-time computation into accuracy on checkable problems; how much RL adds beyond the base model, and where it transfers, is still argued.

Sources

  1. OpenAI, “Learning to Reason with LLMs”, September 2024.
  2. DeepSeek-AI, “DeepSeek-R1”, arXiv:2501.12948, 2025.
  3. Snell, Lee, Xu and Kumar, “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters”, arXiv:2408.03314, 2024.
  4. Yue et al., “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”, arXiv:2504.13837, 2025.