Vol. XIII · Hardware Plate 1 of 9GPU, TPU, accelerator, tensor cores, matrix units, FLOP/s, training timeCE

Accelerators.

Thousands of small multiply–add units, fed from fast memory — and a training run is just their speed against its FLOPs.

Almost all the arithmetic of a Transformer is matrix multiplication (plate V.6), and accelerators are built for exactly that. A data-centre GPU contains over a hundred streaming multiprocessors, each with tensor cores that multiply small blocks of 16-bit or 8-bit numbers in one instruction; Google’s TPUs organise the same work as large systolic arrays. General-purpose arithmetic is a small fraction of their throughput: an H100 delivers about 989 trillion BF16 operations per second in its tensor cores against 67 trillion in ordinary FP32.

Training time follows from three numbers: the run’s FLOPs (about 6ND, plate X.3), the chip’s peak throughput, and the share of that peak actually achieved — the model FLOPs utilisation, typically around 40% (plate XIII.6). The workshop compares five accelerators and computes how long GPT-3’s published 3.14 × 10²³ FLOPs would take on one chip, and how many chips would finish in a month.

Below: left, peak dense BF16 throughput and memory bandwidth of five accelerators, from their spec sheets. Right, for the selected chip, the days GPT-3’s training compute would take on one chip and the chips needed for 30 days, at the chosen utilisation.

What you will learn
Workshop
Spec-sheet figures: dense BF16 tensor throughput and HBM bandwidth (A100 80 GB SXM, H100 SXM, B200 in HGX, MI300X, TPU v5p). Right: GPT-3’s training compute (3.14 × 10²³ FLOPs, Brown et al., Table D.1) at the chosen model FLOPs utilisation.

Throughput, utilisation and FLOPs

GPT-3’s 3.14 × 10²³ training FLOPs would take a single H100 at 40% utilisation about 9,187 days — over five years — while 306 of them would finish in a month. On an A100 it would be 29,121 days; on a B200, 4,038. Each NVIDIA generation shown has raised dense BF16 throughput by about two to three times — H100 over A100 by 3.2 times, B200 over H100 by 2.3 — which is a large part of why training runs of 10²⁵ FLOPs, impossible in 2020, became routine by 2024.

The tensor cores are the reason: on the H100, BF16 tensor throughput is 15 times its FP32 throughput, so any work that cannot be expressed as low-precision matrix multiplication runs at a small fraction of the chip’s speed. That is why so much engineering in this part of the course is about turning everything — attention, communication, optimiser updates — into large matrix multiplies, and keeping the tensor cores fed with data, which is where the limits lie (plates XIII.2–XIII.4).

Peak numbers on spec sheets are often quoted “with sparsity” — assuming half the weights are structured zeros — which doubles them. Training uses dense matrices, so the dense figures are the ones that apply; this course always quotes those.

Terms

Accelerator
A chip specialised for the arithmetic of neural networks, such as a GPU or TPU.
Tensor core
A GPU unit that multiplies small blocks of low-precision matrices in one instruction.
Peak throughput
The maximum FLOP/s a chip can sustain, from its spec sheet.
Model FLOPs utilisation (MFU)
Achieved model FLOP/s divided by peak.
Training time is FLOPs ÷ (chips × peak × utilisation), and peak is the tensor cores’ low-precision matrix throughput.

Sources

  1. NVIDIA A100, H100 and HGX B200 datasheets; AMD Instinct MI300X data sheet; Google Cloud TPU v5p documentation.
  2. Brown et al., “Language Models are Few-Shot Learners”, NeurIPS 2020, Table D.1.