Drills: Hardware.
Five questions at a time from the whole volume, until a spec sheet reads like a forecast.
This plate draws rounds of five from the questions and predictions of the eight plates before it, with the reasoning and source plate after each answer.
The volume’s thread is balance: arithmetic that has outgrown memory, a fast domain inside each server and a slower network outside it, and a utilisation that measures how well a run keeps the arithmetic fed.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Estimate training time from FLOPs and peak.
- Place operations on a roofline.
- Compute intensities and collective times.
- Read spec sheets and MFU claims.
What to carry forward
Accelerators are matrix-multiply engines whose peak sets training time (plate XIII.1); their data comes through a hierarchy whose bandwidth, not arithmetic, often sets the time (plate XIII.2). The roofline decides whether an operation is compute- or memory-bound (plate XIII.3), and for a matrix multiply the intensity is the number of tokens per weight read (plate XIII.4). Interconnect has a fast domain inside a server and a slower network outside (plate XIII.5).
Large runs reach about 40% MFU (plate XIII.6), narrower formats double throughput per halving (plate XIII.7), and four spec-sheet numbers bound what a model can expect (plate XIII.8). Vol. XIV turns to spreading one model over thousands of such chips.
Terms
- Ridge point
- Peak ÷ bandwidth.
- MFU
- Required FLOP/s ÷ peak.
- NVLink
- The fast link inside a server.
- HBM
- Memory beside the chip.