Vol. XIX · Reinforcement Learning for Reasoning Plate 9 of 12entropy collapse, exploration, pass@k, diversity, RLVR, sharpening, mode collapseLE

When Exploration Dies.

First-try accuracy up, any-try accuracy flat: the signature of a policy that has stopped exploring.

A policy learns only from what it samples, so it needs variety: a policy that always gives the same answer cannot discover a better one. RL works against variety, because raising the probability of rewarded answers lowers everything else; the policy’s entropy falls, and with strong optimisation it can collapse to near-deterministic behaviour. Its signature appears in pass@k, the chance that at least one of k samples is right. Pass@1 measures the typical answer; pass@k for large k measures what the policy can reach at all.

Yue and colleagues (2025) compared base and RL-trained reasoning models this way and found RL raised pass@1 while the base model matched or overtook it at large k: RL had mostly concentrated probability on solutions the base could already find. The workshop compares pass@k before and after two GRPO runs on the RL world — a gentle one and a long, aggressive one — and tracks the variety of answers.

Below: left, pass@k on held-out questions before and after training, for the chosen run. Right, the mean number of distinct answers in 16 samples per question, before and after.

What you will learn
Workshop
GRPO from the fine-tuned model, groups of 8 at temperature 1: gentle — 40 steps at step size 0.001; aggressive — 120 steps at 0.002. Pass@k on the 30 held-out sums from n = 16 samples each at temperature 1, with the unbiased estimator. Distinct answers: the mean number of different answers among the 16.

Sharper, not wider

The aggressive run lifts pass@1 from 46% to 58% — its typical answer is better — but pass@16 goes from 70% to 70%: across sixteen tries it reaches no more questions than before, and the variety of its answers has shrunk from 1.93 to 1.53 distinct answers per question. It has concentrated on what it could already sometimes do. The gentle run raises pass@1 to 71% and, here, also pass@16, to 100%: stopping early kept enough variety for the policy to keep finding new right answers.

This is why exploration is a design concern in RL for reasoning. Remedies keep entropy from collapsing — an entropy bonus, a KL penalty to the reference, higher sampling temperature, clipping that limits how fast probabilities can fall — or supply variety from outside, with harder prompts and new data. And it is why pass@k at large k, not just pass@1, belongs in the evaluation of an RL-trained model: a rising pass@1 with a flat pass@k means sharpening, not new capability.

The finding is contested in its general form: other work reports RL extending pass@k on problems far outside the base model’s reach when training runs long enough with fresh problems. Both effects can be real; the measurement that separates them is the one shown here.

Terms

Entropy collapse
A policy’s distribution narrowing until it barely explores.
pass@k
The probability that at least one of k samples is correct.
Sharpening
Concentrating probability on solutions the policy could already find.
Exploration
Sampling varied actions so better ones can be discovered.
Strong RL can raise pass@1 while pass@k stays flat: the policy is sharper, not wider. Keep exploration alive, and measure both.

Sources

  1. Yue et al., “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”, arXiv:2504.13837, 2025.
  2. Chen et al., “Evaluating Large Language Models Trained on Code”, arXiv:2107.03374, 2021 (unbiased pass@k).
  3. Cui et al., “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models”, arXiv:2505.22617, 2025.