Vol. XXIV · Making Inference Cheap Plate 1 of 10quantisation, INT8, INT4, round to nearest, group size, scales, outliers, memoryAE

Quantising Weights.

Fewer bits per weight, fewer bytes per token. The question is how few before the model notices.

Decode reads every weight once per step (plate XXIII.1), so its speed is set by the bytes the weights occupy. Quantisation stores each weight as a small integer and a shared scale: with b bits, a group of weights is divided by s = max|w| / (2^(b−1) − 1), rounded to the nearest integer and multiplied back when used. Llama 3 8B takes 16 GB in BF16, 8 GB at 8 bits and about 4 GB at 4 — on a memory-bound decode, roughly two and four times the tokens per second.

The cost is rounding error, and its size depends on the largest weight in each group, which sets the step for all the others. One scale per output column is the coarsest choice; one scale per group of 32 or 128 weights isolates large weights at the price of storing more scales. The course model micro (1.8 million parameters, four layers; plate I.9) is quantised, pruned and sped up in the page, and its loss is measured on held-out course text it never trained on. The workshop quantises every linear layer of micro by round-to-nearest.

Below: held-out loss of micro after quantising its linear layers, for each bit width, with one scale per column or per group of 32; readouts give the loss increase and the weights’ relative error at the chosen setting.

What you will learn
Workshop
micro’s attention and feed-forward matrices quantised by symmetric round-to-nearest; the embedding stays at 8 bits unless chosen. micro itself is stored with 8-bit weights and one scale per row, and is the reference. Loss: nats per token on the first 1 500 characters of the English and 500 of the Chinese held-out plates, each token predicted once. Relative error: squared error of the quantised weights over their squared sum.

Eight bits free, four bits cheap, two bits broken

Rounding micro’s linear layers to 8 bits raises its held-out loss by 0.0023 nats per token — nothing. At 4 bits the loss rises by 0.327 with one scale per column and 0.132 with one per 32 weights; at 3 bits by 1.366 and 0.673; at 2 bits, with each weight one of three values, by 4.37, and the model no longer works. Smaller groups help at every width, because a single large weight then coarsens the steps of only 31 neighbours instead of a whole column.

Large models behave similarly but with a twist Dettmers and colleagues found: beyond a few billion parameters a handful of activation dimensions carry outliers a hundred times larger than the rest, which wrecks naive 8-bit quantisation of activations and motivated keeping those dimensions in higher precision. For weights alone, 8 bits is routinely lossless and 4 bits with groups the common deployment format — Llama 3 8B in about 4 GB instead of 16. Getting below 4 bits takes the error-aware methods of the next plate.

A weight-only quantised model still computes in 16-bit arithmetic: the integers are expanded on the fly inside the kernel, which is fine for decode because decode waits on memory, not arithmetic.

Terms

Quantisation
Storing numbers with fewer bits as integers times a scale.
Round-to-nearest
Rounding each weight independently to the closest level.
Group size
How many weights share one scale.
Outlier
A value far larger than the rest of its group, which coarsens everyone’s steps.
Quantisation trades bits for rounding error: 8 bits is free, 4 bits with small groups cheap, and below that the method matters.

Sources

  1. Dettmers, Lewis, Belkada and Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, NeurIPS 2022.
  2. Dettmers and Zettlemoyer, “The case for 4-bit precision: k-bit Inference Scaling Laws”, ICML 2023.
  3. Llama Team, AI @ Meta, arXiv:2407.21783, 2024.