Vol. XIX · Reinforcement Learning for Reasoning Plate 8 of 12process reward model, PRM, outcome reward, credit assignment, step-level feedback, PRM800KLE

Rewarding the Steps.

“Wrong answer” says something failed. “Step three is wrong” says what.

An outcome reward scores the whole response by its final answer. A process reward scores each step: was this line of the derivation correct? Lightman and colleagues (2023) collected 800 000 step-level human labels (PRM800K) and trained a process reward model; used to pick the best of many sampled solutions, it solved 78% of a representative subset of the MATH benchmark against 72% for an outcome reward model. The advantage is credit assignment: on a long chain, an outcome reward is rare and shared by every step, right or wrong, while a process reward tells each step how it did.

The workshop isolates the effect in a chain of k steps, each a choice among four actions of which one is right, learned by REINFORCE with a running baseline. The outcome reward pays 1 only if every step is right; the process reward pays each right step 1/k.

Below: left, the share of fully correct chains over training, for outcome and process rewards, at the chosen chain length. Right, the final probability that the whole chain is right, against chain length, for both.

What you will learn
Workshop
A chain of k steps; at each step a tabular softmax policy chooses one of 4 actions, one of them right, starting uniform. 2 000 episodes of REINFORCE (step size 0.5) with a running-mean baseline; the outcome reward credits every step with the chain’s single reward, the process reward credits each step with its own. Success: the share of fully correct chains in each window of 50 episodes.

Credit where it is due

With six steps of four choices, a random chain is right with probability 1 in 4 096. The outcome reward almost never pays early on, and learning stalls: after 2 000 episodes the whole chain is right 0.0% of the time. The process reward pays for every right step from the first episode and reaches 99.5%. At four steps both succeed (100% and 100%); at eight the outcome reward gets nowhere (0.0%) while the process reward still learns (99.4%).

Real reasoning chains are not a fixed sequence of independent choices, and outcome-reward RL does work on them — because the policy starts from a model that already succeeds often enough for rewards to arrive (plate XIX.1). The case for process rewards is strongest where success is rare or chains are long. Their cost is the judging: step labels from people are expensive, step labels from models can be wrong or gamed, and a step can look right while leading nowhere. DeepSeek-R1’s authors reported trying process reward models and abandoning them for exactly these reasons; others generate step labels automatically by checking which steps tend to lead to correct answers.

Uesato and colleagues found outcome- and process-based supervision gave similar final-answer accuracy, but process supervision cut the rate of right answers reached through wrong reasoning — the distinction between a right answer and a right derivation.

Terms

Outcome reward
A reward for the final answer only.
Process reward
A reward for each step of the reasoning.
Credit assignment
Deciding which actions in an episode earned its reward.
PRM
Process reward model: a model trained to judge individual steps.
An outcome reward is rare and shared by every step; rewarding steps solves credit assignment on long chains — if the steps can be judged reliably.

Sources

  1. Lightman et al., “Let’s Verify Step by Step”, ICLR 2024 (PRM800K; 78% against 72% on a MATH subset).
  2. Uesato et al., “Solving math word problems with process- and outcome-based feedback”, arXiv:2211.14275, 2022.
  3. Wang et al., “Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations”, ACL 2024.