Vol. XIII · Hardware Plate 2 of 9memory hierarchy, HBM, SRAM, L2 cache, shared memory, registers, PCIe, bandwidth, capacityCE

The Memory Hierarchy.

Kilobytes next to the arithmetic, megabytes on the chip, gigabytes beside it, and a narrow pipe to everything else.

An H100 has 132 streaming multiprocessors. Each holds a 256 KB register file and up to 228 KB of shared memory — fast SRAM that a program manages itself — and all share a 50 MB L2 cache. Beside the die sit 80 GB of high-bandwidth memory (HBM), stacked DRAM read at 3.35 TB/s. Beyond that is the host computer’s memory, reached over PCIe at about 64 GB/s each way. Each step down is larger and slower.

A model’s weights and the activations of a training step live in HBM, because nothing smaller holds them; every use of them must stream them up the ladder to the tensor cores. The speed of a computation therefore depends on how many times its data crosses the HBM boundary, which is the subject of the roofline (plate XIII.3) and of kernels such as FlashAttention (plate XVI.4). The workshop puts the levels on one scale and asks where a given tensor fits and how long it takes to move.

Below: left, the capacity of each level of an H100’s memory hierarchy, log scale. Right, how long it takes to read a tensor of the chosen size from HBM, over NVLink and over PCIe.

What you will learn
Workshop
H100 SXM, from NVIDIA’s architecture whitepaper and datasheet: registers 256 KB and shared memory 228 KB per SM (× 132 SMs), L2 cache 50 MB, HBM3 80 GB at 3.35 TB/s, NVLink 900 GB/s total (450 GB/s each way), PCIe Gen5 x16 64 GB/s each way. Right: time to move a tensor of the chosen size.

Every crossing costs

Llama 3 8B’s weights in BF16 are about 16 GB: they fit in HBM and nowhere smaller, and reading all of them once takes 4.79 × 10⁻³ seconds at 3.35 TB/s — the floor on the time to generate one token for one user on one GPU (plate XXIII.1). Over PCIe the same read would take 0.3 seconds, 52 times longer, which is why models are kept entirely in accelerator memory and why offloading to the host is a last resort. A tensor of the default size here, 1.6 GB, fits only in HBM; one of a few hundred kilobytes fits in a multiprocessor’s shared memory, where it can be reused at far higher bandwidth.

This is the central fact of accelerator performance: the arithmetic units can consume data much faster than HBM can supply it, so a computation is fast only if it reuses each byte many times once it is on the chip. Matrix multiplication can — every element of a weight matrix is used once per token in the batch — while element-wise operations cannot. The next two plates turn that into numbers.

HBM bandwidth has grown more slowly than tensor throughput: from A100 to H100 throughput rose about threefold, bandwidth by about 1.6 times. Each generation therefore needs more reuse per byte to stay busy.

Terms

HBM
High-bandwidth memory: stacked DRAM beside the accelerator die; tens of GB at a few TB/s.
Shared memory
Programmer-managed on-chip SRAM in each multiprocessor.
L2 cache
Shared on-chip cache between HBM and the multiprocessors.
PCIe
The bus to the host computer; far slower than HBM or NVLink.
Each level down the hierarchy is larger and slower; models live in HBM, and speed depends on reusing data once it is on the chip.

Sources

  1. NVIDIA, “NVIDIA H100 Tensor Core GPU Architecture” whitepaper, 2022; NVIDIA H100 datasheet.