Vol. XVI · Kernels and Efficiency Plate 6 of 8quantisation, INT8, INT4, FP8, weight-only quantisation, dequantisation, decode, prefill, tensor coresCE

Low-Precision Matrix Multiply.

Fewer bits per weight speeds up reading; fewer bits per multiply speeds up arithmetic. Which one helps depends on the batch.

There are two kinds of low-precision multiply. Weight-only quantisation stores weights in INT8 or INT4 with a scale per group, reads them in that form, widens them to BF16 inside the kernel, and multiplies in BF16. Weight-and-activation formats such as FP8 (plate XI.6) feed both operands to the tensor cores in 8 bits, which on an H100 run at twice the BF16 rate.

Which helps depends on where the multiply sits on the roofline (plate XIII.3). Decoding a few tokens reads every weight to do a few FLOPs with it: memory-bound, so halving or quartering the weight bytes nearly halves or quarters the time. Prefill and training multiply thousands of tokens by each weight: compute-bound, so only faster arithmetic helps — and the widening costs extra work every time the weights are reloaded. The workshop times one Llama 3 8B MLP projection (4 096 → 14 336) in all four formats.

Below: left, time per token for the projection against the batch, in each format. Right, the time of each format at the chosen batch.

What you will learn
Workshop
One 4 096 × 14 336 projection for b tokens on an H100 by the roofline: weights read once in the format’s width, BF16 activations in and out; peak 989 TFLOP/s for BF16 and the weight-only formats, 1 979 for FP8. Weight-only kernels widen each weight once per 128 tokens at about 4 operations each on the non-tensor units (2 × the FP32 rate). Scales and zero points are ignored.

Fewer bytes or faster arithmetic

Decoding one token, the BF16 projection is limited by reading 117 MB of weights; storing them in INT4 makes it 3.33 times faster — not quite four, because the activations and the widening still cost something. At 8 192 tokens the same INT4 kernel is 0.90 times the speed of BF16, a slowdown: the multiply is compute-bound, the weights’ bytes no longer matter, and widening them for every block of 128 tokens is extra work. FP8 is the reverse: 2.00× at one token, where it only halves the bytes, and 2.00× at 8 192, where it doubles the arithmetic rate.

This is why deployments use weight-only INT4 or INT8 for latency-bound decoding on small batches, and FP8 for high-throughput serving and for training (plate XI.6), where the tensor cores are the limit. Both trade accuracy for speed in ways that must be measured on the model: per-group scales, outlier handling and, for FP8, per-tensor or per-block scaling (plate XI.6’s microscaling) decide whether a quantised model keeps its quality.

The batch where a format stops helping is the ridge point in disguise: weight-only quantisation helps while tokens per weight read are well below the H100’s roughly 295 FLOPs per byte, and stops once the batch passes it.

Terms

Weight-only quantisation
Storing weights in few bits and widening them for a higher-precision multiply.
Dequantisation
Converting quantised values back to a wider format, using scales.
FP8 multiply
A tensor-core multiply with both operands in 8-bit floating point.
Group scale
One scale shared by a small group of quantised weights.
Weight-only quantisation speeds memory-bound decoding by reading fewer bytes; FP8 speeds compute-bound work by faster arithmetic. The batch decides which matters.

Sources

  1. Frantar et al., “GPTQ”, ICLR 2023; Lin et al., “AWQ”, MLSys 2024 (weight-only quantisation).
  2. Micikevicius et al., “FP8 Formats for Deep Learning”, arXiv:2209.05433, 2022.
  3. NVIDIA H100 specifications (FP8 tensor throughput twice BF16).