Drills: Clusters.
Five questions at a time from the whole volume, until a training report reads like a ledger.
This plate draws rounds of five from the questions and predictions of the eight plates before it, with the reasoning and source plate after each answer.
The volume’s thread is attrition: at sixteen thousand GPUs, failures, slow devices, silent errors and bookkeeping slips are certainties, and a run’s success is measured by how little of its time they take.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Estimate failure rates and checkpoint intervals.
- Explain stragglers and silent corruption.
- Design resumption and monitoring.
- Recompute a report’s cost and utilisation.
What to carry forward
A synchronous job fails every M/n (plate XV.1), so checkpoint every √(2δM) and make restarts fast (plate XV.2). Jitter and slow devices cost σ√(2 ln n) or more (plate XV.3), and silent corruption can be loud, harmless or invisible to everything but a checksum or replay (plate XV.4).
The data loader must rewind with the model (plate XV.5), the model and the machine must be watched separately (plate XV.6), and costs and utilisation follow from GPU-hours and their assumptions (plates XV.7–XV.8). Vol. XVI looks inside a single step, at the kernels where the time goes.
Terms
- MTBF
- Mean time between failures.
- Young–Daly
- τ* = √(2δM).
- SDC
- Silent data corruption.
- PUE
- Facility power ÷ IT power.