Vol. XIII · Hardware Plate 7 of 9number formats, FP64, FP32, TF32, BF16, FP8, FP4, tensor core throughput, bytes per valueCE

Number Formats in Silicon.

Fewer bits per number, more numbers per second — and fewer bytes to fetch for each one.

The H100’s tensor cores deliver about 67 dense TFLOP/s in FP64, 495 in TF32 (FP32 range with a 10-bit mantissa), 989 in BF16 and FP16, and 1 979 in FP8. Each halving of the width roughly doubles throughput, because the multipliers are smaller and more fit in the same silicon. Blackwell-generation chips add 4-bit floating point for inference at about double the FP8 rate.

Narrower formats also halve the bytes per value, so memory-bound operations speed up too, and a model’s weights take less memory. The limits are numerical (plates XI.6–XI.7): too few bits lose the precision or range training needs. The workshop shows one H100’s throughput by format, and what each format would do to the time of a large matrix multiply and the memory of a model’s weights.

Below: left, H100 dense tensor throughput by number format. Right, for the chosen model size, weight memory and the time of its forward FLOPs per token on one H100, by format.

What you will learn
Workshop
H100 SXM dense tensor throughput by format from NVIDIA’s datasheet (FP64 67, TF32 495, BF16 989, FP8 1 979 TFLOP/s). Right: weight memory and the compute-bound time of 2N FLOPs (one token’s forward pass) at each format’s peak.

Halve the bits, double the rate

On an H100, BF16 runs 2.0 times faster than TF32 and FP8 2.0 times faster than BF16 — the doubling per halving of width that tensor cores are designed for. For a 70-billion-parameter model the weights take 141.6 GB in BF16, which does not fit on one 80 GB H100, and 70.8 GB in FP8, which does. A 405-billion-parameter model takes 796.2 GB in BF16: even at FP8 it needs several GPUs just to hold its weights (Vol. XIV).

The choice of format is made operation by operation, not for a whole model: matrix multiplications in BF16 or FP8, accumulations and optimiser states in FP32, sensitive layers kept wider (plates XI.6–XI.7), and for inference weights often in 8 or 4 bits (Vol. XXIV). Chips follow the formats that prove useful — FP8 entered tensor cores with Hopper, FP4 with Blackwell — and each new format becomes worth the engineering effort only when it brings this doubling.

TF32 is a tensor-core format, not a storage format: values are stored as FP32 and the multiplier uses only 10 mantissa bits. Frameworks can use it silently for FP32 matrix multiplies, which speeds them up at a small loss of precision.

Terms

TF32
A tensor-core format with FP32’s range and a 10-bit mantissa.
FP8
Eight-bit floating point, E4M3 or E5M2 (plate XI.7).
FP4
Four-bit floating point, supported by Blackwell tensor cores for inference.
Bytes per value
Bits ÷ 8; sets weight memory and bandwidth per value.
Tensor-core throughput roughly doubles with each halving of the number format, and so do memory and bandwidth savings; formats are chosen per operation.

Sources

  1. NVIDIA H100 datasheet and H100 architecture whitepaper.
  2. NVIDIA Blackwell architecture technical brief, 2024 (FP4 tensor cores).