Vol. XV · Running a Training Cluster Plate 1 of 9failures, mean time between failures, MTBF, reliability, interruptions, hardware faults, Llama 3 Table 5CE

Failures at Scale.

A part that fails once in six years, multiplied by sixteen thousand, fails every three hours.

Synchronous training makes a cluster as fragile as its weakest part: every step needs every device, so one failed GPU, one bad cable or one crashed process stops all of them until the job restarts. If each of n devices fails independently with mean time between failures M, the job fails on average every M/n. The Llama 3 team published the record of a 54-day snapshot of 405B pre-training on up to 16 384 H100s: 466 interruptions, 47 planned for maintenance and 419 unexpected.

Their Table 5 sorts the unexpected ones by root cause. GPUs — faulty chips, their HBM memory, SRAM, system processor and thermal sensors — account for most; software bugs, network switches and cables, and host maintenance for much of the rest; six were silent data corruption (plate XV.4). The workshop turns the snapshot into a per-device failure rate and asks what it means for clusters of other sizes.

Below: left, the mean time between job interruptions against cluster size for the chosen per-device reliability, with Llama 3’s snapshot marked. Right, the 419 unexpected interruptions by root cause.

What you will learn
Workshop
Each device fails independently at a constant rate, with the per-device mean time between failures set by the slider; its default is what Llama 3’s snapshot implies, 54 days × 24 h × 16 384 GPUs ÷ 419 unexpected interruptions, attributing every interruption to some GPU’s share of the cluster. Right: the interruptions of Table 5, with the seven smallest host and dependency rows combined.

Every three hours

In 54 days the Llama 3 405B job was interrupted unexpectedly 419 times, once every 3.1 hours. Spread over 16 384 GPUs that is one failure per GPU every 5.8 years — individually very reliable hardware. GPU-related causes alone were 64% of the total, and confirmed or suspected hardware about 78%. At this rate a cluster of 1 024 GPUs would be interrupted every 50 hours, and one of 131 072 every 0.39 hours — 62 times a day.

Nothing about this is negligence: it is arithmetic. The defences are in the plates that follow — restart quickly from frequent checkpoints (plate XV.2), find slow but working hardware (plate XV.3), catch errors that do not crash (plate XV.4) — and automation, because a person cannot be paged eight times a day. Meta reports that only three of the interruptions needed significant manual intervention; everything else was detected, isolated and restarted by software.

The per-device figure is an average over everything in the cluster that can stop a job — GPUs, hosts, switches, software — attributed per GPU. It is not a GPU vendor’s reliability rating, and real failure rates are higher early in a cluster’s life and late in its components’.

Terms

MTBF
Mean time between failures.
Synchronous job
A job whose every step needs every device, so any failure stops it.
Interruption
A stop of the training job, planned or unexpected, requiring a restart.
Root cause
The component or fault an interruption is finally attributed to.
A synchronous job fails whenever any of its n devices does, so its failure interval is the device’s divided by n — every few hours at sixteen thousand GPUs.

Sources

  1. Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024, section 3.3.4 and Table 5.