Vol. XI · Optimisation at Scale Plate 6 of 10mixed precision, FP32, BF16, FP16, loss scaling, master weights, underflow, roundingCE

Mixed Precision.

Multiply in 16 bits, accumulate and update in 32 — and, in FP16, scale the loss so tiny gradients survive.

A 32-bit float (FP32) has 8 exponent bits and 23 mantissa bits. The two 16-bit formats split their bits differently. FP16 has 5 exponent bits and 10 mantissa bits: good precision, but a narrow range — numbers below about 6 × 10⁻⁸ underflow to zero and above 65 504 overflow. BF16 keeps FP32’s 8 exponent bits and has only 7 mantissa bits: the same range as FP32, but only two or three significant decimal digits.

Mixed-precision training uses each where it is safe. Matrix multiplications run in 16 bits — accelerators run them several times faster than in FP32 (plate XIII.7) — while a master copy of the weights and the optimiser states stay in FP32, because an update far smaller than the weight would be lost to rounding in 16 bits. With FP16, gradients that are too small would underflow, so the loss is multiplied by a large loss scale before backpropagation and the gradients divided by it afterwards. BF16, with FP32’s range, needs no scaling, which is why it became the default for large models.

Below: left, a spread of gradient magnitudes and the share that underflows or overflows in the chosen 16-bit format, with the chosen loss scale. Right, a weight of 1.0 receiving many small updates, stored in BF16 or with an FP32 master copy.

What you will learn
Workshop
Left: 4 000 gradient values whose magnitudes are spread log-normally around the chosen centre (one decade standard deviation), multiplied by the loss scale and rounded to the chosen format; shares lost to underflow (zero) and overflow (infinity). Right: a weight of 1.0 receiving 1 000 updates of +δ each, stored in BF16 throughout or kept in an FP32 master copy.

Range, precision and where each matters

With gradients centred on 10⁻⁷, FP16 without scaling loses 30% of them to underflow: they become exactly zero, and those weights stop learning. A loss scale of 2¹⁶ shifts them into range and only 0.0% underflow; scale too far with large gradients and they overflow instead (29% at 2²⁴ with gradients near 10⁻³), which is why practical loss scaling is dynamic — raised until an overflow appears, then lowered. BF16 needs none of this: with the same gradients it loses 0.0%, because it has FP32’s exponent range.

The price of BF16 is precision, and the right panel shows where it bites. A weight of 1.0 receiving updates of 10⁻³ in BF16 never moves — each update is below half the spacing of representable numbers near 1, so every addition rounds back to 1.0 — while the FP32 master copy reaches 2.000. Updates of 10⁻² register at first, but the spacing of BF16 numbers doubles with every power of two, and the weight stalls at 4.000 once the spacing outgrows the update, against 11.000 in FP32. Hence the standard recipe: BF16 for the matrix multiplications and activations, FP32 for the master weights, the optimiser states and the accumulators that sum many small numbers.

The FP32 master weights and two Adam states cost 12 bytes per parameter on top of the 16-bit copies, which is why the optimiser state dominates training memory (plate XIV.1) and why sharding it across devices (plate XIV.5) matters.

Terms

Mantissa bits
Bits for the significant digits; they set precision.
Exponent bits
Bits for the power of two; they set range.
Loss scaling
Multiplying the loss before backpropagation so that small gradients stay representable in FP16.
Master weights
An FP32 copy of the weights to which updates are applied.
16-bit arithmetic is fast but lossy: FP16 loses range, BF16 loses precision, so updates go to FP32 master weights and FP16 gradients need loss scaling.

Sources

  1. Micikevicius et al., “Mixed Precision Training”, ICLR 2018 (FP32 master weights and loss scaling).
  2. Kalamkar et al., “A Study of BFLOAT16 for Deep Learning Training”, arXiv:1905.12322, 2019.