Vol. XIII · Hardware Plate 8 of 9spec sheet, accelerator comparison, peak FLOP/s, memory capacity, bandwidth, ridge point, decode speed, training throughputCE

Reading a Spec Sheet.

Peak, capacity, bandwidth, link — four numbers, and three things a model can do with them.

A spec sheet lists many numbers, but four decide most of what matters for language models. Peak dense tensor FLOP/s in the training format sets training throughput. Memory capacity sets how much of a model, and of its key–value cache, fits on one chip. Memory bandwidth sets the speed of every memory-bound operation, and above all of generating text, where each token reads all the weights. Interconnect bandwidth sets how well many chips work as one.

Their ratios matter as much as their values. Peak ÷ bandwidth is the ridge point (plate XIII.3); capacity ÷ bandwidth is the time to read the whole memory once, which bounds decode speed. The workshop derives these for five accelerators and asks what each would do for a model of the chosen size: whether its BF16 weights fit, how many tokens per second one chip could train at 40% MFU, and an upper bound on tokens per second when generating for a single user.

Below: left, the four numbers and derived ratios for the selected accelerator. Right, for the chosen model size on each accelerator: training tokens per second per chip at 40% MFU and the bandwidth bound on single-user decode speed.

What you will learn
Workshop
Spec-sheet values as in plate XIII.1. Left: the selected chip’s figures and derived ratios. Right: for a model of the chosen size in BF16, training tokens per second per chip at 40% MFU (6N FLOPs per token) and the decode bound for one user (each token reads all 2N bytes of weights), for each accelerator.

What the four numbers promise

For an 8-billion-parameter model on an H100, the spec sheet promises about 8,301 training tokens per second per chip at 40% MFU, and at most 211 tokens per second when generating for one user, because each token must read all 7.94 billion weights at 2 bytes each from HBM. The ridge point of 295 FLOPs per byte and a full-memory read time of 23.9 ms summarise the balance. A B200 raises the decode bound to 504 with its higher bandwidth; an MI300X, with 192 GB at 5.3 TB/s, to 334, and its capacity holds larger models on fewer chips.

Real systems fall short of these bounds — decode reaches perhaps half to three-quarters of the bandwidth bound, training around 40% of peak — but the bounds are the right first estimate and the right check on a vendor’s or a paper’s claim. They also show where each chip is strong: training throughput tracks peak FLOP/s, single-user generation tracks bandwidth, and the size of model a chip can serve alone tracks capacity. Which matters most depends on whether one is training, serving many users, or serving one.

Spec-sheet peaks are measured on ideal matrix shapes at full clock speed. Sustained throughput under power and thermal limits can be lower, which is one more reason achieved MFU sits well below 100%.

Terms

Peak FLOP/s
The spec-sheet maximum for dense tensor operations in a given format.
Memory capacity
HBM per chip; sets what fits.
Memory bandwidth
HBM read and write rate; sets memory-bound speed.
Decode bound
Bandwidth ÷ bytes of weights: the most tokens per second one user can get.
Peak sets training, bandwidth sets single-user generation, capacity sets what fits — and their ratios give the first estimate of any claim.

Sources

  1. NVIDIA A100, H100 and HGX B200 datasheets; AMD Instinct MI300X data sheet; Google Cloud TPU v5p documentation.