Vol. XIX · Reinforcement Learning for Reasoning Plate 5 of 12GRPO, group-relative policy optimisation, advantage, group baseline, DeepSeekMath, DeepSeek-R1, value-free RLLE

Group-Relative Advantages.

Ask the same question several times; reward the answers that beat their siblings.

Group-relative policy optimisation (GRPO), introduced with DeepSeekMath (Shao et al., 2024) and used to train DeepSeek-R1, removes PPO’s value network. For each prompt it samples a group of G answers, computes their rewards, and gives each answer the advantage (r − mean)/std over its own group. An answer is pushed up if it did better than its siblings and down if worse; the baseline is specific to the prompt, which a single batch-wide mean is not. The rest is PPO’s clipped objective, usually with a KL penalty to the reference.

The design has a blind spot that matters in practice: if all G answers to a prompt get the same reward — all right on an easy prompt, all wrong on a hard one — every advantage is zero and the prompt contributes nothing. The workshop trains the RL world with GRPO at a fixed budget of 64 answers per step, split into groups of the chosen size, and counts the groups that carried no signal.

Below: left, training reward and held-out accuracy; right, the share of groups whose answers were all right or all wrong, step by step, for the chosen group size.

What you will learn
Workshop
GRPO from the fine-tuned model: 64 answers a step, as 64/G prompts with G answers each at temperature 1; advantage normalised within each group (zero if the group’s rewards are all equal); one epoch, step size 0.001, 40 steps. Dead groups: those with all rewards equal.

Better than its siblings

With groups of 8, 63% of groups had all answers right or all wrong and taught nothing, and held-out accuracy ends at 73%. With groups of 2 — 32 prompts a step, two answers each — 87% of groups carried no signal, because two answers to a prompt the model mostly gets right are usually both right. With groups of 16, 53%: a larger group is more likely to contain both a success and a failure, at the price of fewer prompts per step.

The waste grows as training succeeds: the better the policy gets, the more prompts it always answers correctly, and the fewer groups contain a failure to learn from. Later systems answer this directly — DAPO keeps sampling until a batch is filled with groups that have mixed rewards, and curricula move towards prompts the policy sometimes fails. GRPO’s other choices have been questioned too: dividing by the group’s standard deviation over-weights prompts that are almost solved, and averaging the loss per reply rather than per token favours short answers; variants such as Dr. GRPO remove both.

GRPO’s normalisation makes the advantage scale-free: a reward of 0 or 1 and a reward of 0 or 100 give the same updates, which is convenient for mixing tasks and dangerous when a tiny reward difference becomes a full unit of advantage.

Terms

GRPO
Group-relative policy optimisation: advantages from a group of answers to the same prompt.
Group baseline
The mean reward of the group, used in place of a value function.
Dead group
A group whose answers all earn the same reward and so carry no signal.
Dynamic sampling
Resampling until groups with mixed rewards fill the batch.
GRPO scores each answer against its own group — no value network — and a group that is all right or all wrong teaches nothing.

Sources

  1. Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models”, arXiv:2402.03300, 2024.
  2. DeepSeek-AI, “DeepSeek-R1”, arXiv:2501.12948, 2025.
  3. Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale”, arXiv:2503.14476, 2025 (dynamic sampling of non-uniform groups).