Vol. XVIII · Learning From Preferences Plate 4 of 11RLHF, PPO, policy, reward model, KL penalty, reference model, rollouts, advantageLE

RLHF With PPO.

Sample, score, penalise the drift, update — and repeat until the policy earns what the reward model will pay.

The InstructGPT recipe has three stages: supervised fine-tuning (Vol. XVII), a reward model from comparisons (plate XVIII.3), and reinforcement learning. In the third, the policy — initialised as the fine-tuned model — generates responses to prompts; each is scored by the reward model, minus β times the log-ratio of the policy’s probability to the frozen reference’s, which keeps the policy from drifting into text the reward model never saw. Proximal Policy Optimisation (Schulman et al., 2017) then raises the probability of responses that scored above average, clipping each update so the policy changes only a little per step.

For a reward r and strength β, the best policy is known in closed form: π*(y) ∝ π_ref(y) exp(r(y)/β). PPO is a way of reaching it by sampling, because for a language model the sum over all possible responses cannot be computed. In the preference world it can, so the workshop runs PPO on the policy’s six parameters and compares its path with the exact optimum.

Below: left, the policy’s expected reward, true quality and KL from the reference over PPO steps, with the exact optimum dashed. Right, the probability of each response type before and after.

What you will learn
Workshop
Reward model fitted to 1 000 comparisons from raters with bias 1. Policy π(y) ∝ π_ref(y) exp(θ·f(y)). Each PPO step: 64 rollouts, reward r(y) − β log(π/π_ref), advantages normalised over the batch, 4 epochs of gradient ascent on the clipped surrogate at step size 0.3, clip 0.2. Curves are exact expectations of the current policy; dashed lines the closed-form optimum.

Sampling towards a known answer

At β ≈ 0.3, PPO starts from the reference policy — reward 0.90, true quality 1.20 — and in 60 steps reaches reward 1.58 at a KL of 0.74 nats, within 0.012 of the reward the closed form predicts. True quality rises with it, to 1.82, because at this β the policy mostly stops giving wrong answers. The bars show where the probability went: away from wrong and hedged answers, towards the correct ones the reward model likes — including the long, flattering one.

Real RLHF has everything this picture leaves out: responses are sampled token by token, a value network estimates expected reward for the advantage, the reward is given only at the end of a response, and four large models — policy, reference, reward and value — must sit in memory together. PPO’s clipping and the KL penalty are what keep it stable. But its target is the same: the policy that trades reward against distance from the reference at the rate β sets — and what that policy does depends entirely on what the reward model rewards.

β is usually tuned so the final KL lands at a few nats; InstructGPT used a per-token KL coefficient of 0.02 and also mixed in pre-training gradients to limit the alignment tax.

Terms

RLHF
Reinforcement learning from human feedback: optimising a policy against a learned reward model.
Policy
The model being trained, viewed as a distribution over responses.
PPO
Proximal Policy Optimisation: policy-gradient updates with a clipped probability ratio.
Reference model
The frozen fine-tuned model the KL penalty measures distance from.
RLHF maximises reward minus β times the KL from the reference; PPO samples its way to the policy π_ref · e^{r/β}.

Sources

  1. Ouyang et al., NeurIPS 2022.
  2. Schulman et al., “Proximal Policy Optimization Algorithms”, arXiv:1707.06347, 2017.
  3. Christiano et al., “Deep reinforcement learning from human preferences”, NeurIPS 2017.
  4. Rafailov et al., NeurIPS 2023, section 4 (closed-form KL-regularised optimum).