Vol. XXX · The Research Frontier Plate 2 of 11scaling RL, RL compute, sigmoid scaling, asymptote, compute efficiency, ScaleRL, post-training computeCE

Scaling Reinforcement Learning.

Pretraining had its power laws. RL is getting its own curves — with a ceiling.

In 2023, post-training was a small fraction of a model’s compute. By 2025, developers were describing reinforcement learning runs on a scale comparable to pretraining (reported, mostly in announcements without full detail), and the question became whether RL’s returns are predictable. Khatri and colleagues (October 2025, ICLR 2026) spent over 400,000 GPU-hours fitting validation reward R against RL compute C with a saturating curve R(C) = R₀ + (A − R₀)/(1 + (C_mid/C)^B): an asymptote A that the recipe can reach, an efficiency exponent B, and a midpoint C_mid (reported).

Their finding is that many design choices — loss aggregation, normalisation, curriculum, off-policy corrections — move B and C_mid (how fast the curve rises) but not A (where it ends), while others change the ceiling; so a recipe should be judged by its fitted ceiling, not by which is ahead early. The workshop draws two such curves with the chosen parameters and shows how a run that leads early can lose at scale.

Below: two RL scaling curves — a recipe with a higher ceiling and one with higher early efficiency — against compute; readouts give where they cross.

What you will learn
Workshop
Illustrative curves of the published functional form, not fitted values: R₀ = 0.2 for both; recipe 1 with ceiling A₁ and efficiency B₁ = 1 and midpoint 1 000 GPU-hours; recipe 2 with ceiling 0.6 and the chosen efficiency and midpoint. Compute on a logarithmic axis.

Judge a recipe by its ceiling

With the default curves, recipe 2 is ahead at 100 GPU-hours (0.400 against 0.250) because it rises faster, and a team comparing short runs would choose it; but its ceiling is lower, and recipe 1 overtakes at about 3,162 GPU-hours and ends at 0.749 against 0.600. The published study’s point is exactly this: fit the curve from runs up to a moderate budget, compare asymptotes, and only then scale — the authors validated their recipe on a single run of 100,000 GPU-hours (reported).

What is established is narrower than for pretraining: RL performance rises with compute on verifiable tasks and eventually saturates for a given recipe and task set. What is contested or unknown: whether the ceiling can be raised indefinitely by more diverse environments (Vol. XXVII), how RL compute should be split against pretraining compute for a fixed budget, and whether the curves for agentic, long-horizon tasks behave like those for maths.

The sigmoid differs from pretraining’s power law in one important way: it has a ceiling. Power laws in pretraining also bend eventually, but within the studied range they did not.

Terms

Asymptote
The reward a recipe approaches with unlimited compute.
Compute efficiency
How quickly reward rises with compute.
Recipe
The full set of RL design choices.
Extrapolation
Predicting large runs from fitted small ones.
RL returns follow saturating curves with a ceiling; compare recipes by fitted ceilings, because the early leader can lose at scale.

Sources

  1. Khatri et al., “The Art of Scaling Reinforcement Learning Compute for LLMs”, arXiv:2510.13786, 2025.
  2. DeepSeek-AI, “DeepSeek-R1”, arXiv:2501.12948, 2025.
  3. Kimi Team, “Kimi k1.5: Scaling Reinforcement Learning with LLMs”, arXiv:2501.12599, 2025.