Vol. XIX · Reinforcement Learning for Reasoning Plate 4 of 12PPO, clipping, probability ratio, epochs, off-policy, trust region, value function, GAELE

PPO in Detail.

Several updates from one batch, each held on a short leash by the clipped ratio.

Sampling is the expensive part of RL for language models, so PPO (Schulman et al., 2017) squeezes several gradient steps out of each batch. After the first step the policy π_θ differs from the policy π_old that produced the samples, so each token’s contribution is weighted by the ratio ρ = π_θ(y_t)/π_old(y_t). Left alone, repeated steps could push some ratios far from 1 and the policy far from where its data came from. PPO clips: once ρ leaves [1 − ε, 1 + ε] in the direction the advantage favours, that token stops contributing gradient.

Full PPO for language models also learns a value function — a second network predicting expected reward from each prefix — and uses generalised advantage estimation (GAE) to give each token its own advantage. With a single reward at the end and a mean baseline, as here, the per-token advantages are all equal and the value function is not needed; that simplification is what GRPO formalises (plate XIX.5). The workshop runs PPO on the RL world with a chosen number of epochs per batch and clip range.

Below: left, the reward during training and held-out accuracy, for the chosen epochs and clip range. Right, the share of tokens whose update was clipped, step by step.

What you will learn
Workshop
PPO from the fine-tuned model: 8 prompts a step, 8 samples each at temperature 1, advantage = reward minus the batch mean (the same for every token of a reply), the clipped objective optimised for the chosen number of epochs at step size 0.001, 40 steps. Clipped share: the fraction of answer tokens whose ratio was outside the clip range in the favoured direction, averaged over the epochs of a step.

Reuse, held on a leash

With one epoch per batch the ratio is always 1 and nothing is clipped (0.0%): PPO reduces to the baseline policy gradient. With four epochs and ε = 0.2, 7.1% of tokens are clipped on average — the later epochs have pushed those tokens’ probabilities as far as PPO allows from this batch — and the run ends at training reward 0.89 and held-out accuracy 70%. Eight epochs clip 14.9%; widening the range to 0.5 clips 4.9%, letting each batch move the policy further.

The clipped share is a practical gauge: near zero, the epochs are wasted on a policy that barely moves; very large, the batch is being over-used and the policy is drifting from its own data. Production PPO for language models adds more machinery — a value network as large as the policy, GAE with a discount and λ, KL control against the reference (plate XVIII.5), reward normalisation — and its many details matter enough that whole papers list them. Much of the recent shift to GRPO and RLOO is about removing the value network and its memory.

The ratio is computed per token, but the reward is per reply. Clipping a token stops the gradient through that token only; the reply’s other tokens still update — one reason token-level and sequence-level variants of PPO-like objectives behave differently.

Terms

Probability ratio
π_θ(y_t)/π_old(y_t): how much the policy has changed for a sampled token.
Clipping
Stopping the gradient once the ratio leaves [1 − ε, 1 + ε] in the favoured direction.
Value function
A network predicting expected reward from a state, used as a baseline.
GAE
Generalised advantage estimation: blending value predictions into per-token advantages.
PPO reuses each batch for several epochs and clips the probability ratio so the policy cannot move far from the one that sampled.

Sources

  1. Schulman, Wolski, Dhariwal, Radford and Klimov, “Proximal Policy Optimization Algorithms”, arXiv:1707.06347, 2017.
  2. Schulman et al., “High-Dimensional Continuous Control Using Generalized Advantage Estimation”, ICLR 2016.
  3. Huang et al., “The N+ Implementation Details of RLHF with PPO”, 2024.