Drills: Parallelism.
Five questions at a time from the whole volume, until a layout reads like a budget.
This plate draws rounds of five from the questions and predictions of the ten plates before it, with the reasoning and source plate after each answer.
The volume’s thread is division: the memory a model needs, divided among devices in different ways, each paid for in communication of a different kind over links of different speeds.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Total the memory to train a model.
- Match each parallelism to its collective.
- Compute bubbles and communication ratios.
- Check a layout against memory and utilisation.
What to carry forward
Training needs 16 bytes per parameter plus activations (plate XIV.1); recomputation trades activations for compute (plate XIV.2). Four collectives carry all the traffic, and a ring all-reduce costs about the same per device however many join (plate XIV.3). Data parallelism scales while compute hides the all-reduce (plate XIV.4), and ZeRO removes its redundancy (plate XIV.5).
Tensor parallelism splits every layer and stays on NVLink (plate XIV.6); pipelines cross servers at the cost of a bubble (plate XIV.7); context parallelism splits long sequences (plate XIV.8); expert parallelism spreads experts with all-to-alls (plate XIV.9). A layout multiplies them (plate XIV.10). Vol. XV turns to keeping such a cluster running for months.
Terms
- All-reduce
- Sum everywhere: reduce-scatter plus all-gather.
- ZeRO
- Shard the model states.
- Bubble
- (p − 1)/(m + p − 1).
- Layout
- t × p × dp = GPUs.