Vol. XV · Running a Training Cluster Plate 3 of 9stragglers, synchronous training, tail latency, expected maximum, slow node, thermal throttling, diurnal variationCE

Stragglers.

The step waits for the slowest of sixteen thousand. One slow GPU is a slow cluster.

Every synchronous step ends with a collective that waits for every device. If each device’s step time varies a little — from thermal throttling, a busy host, a congested link — the step takes as long as the slowest of them, and the expected maximum of n random times keeps growing with n: for small Gaussian-like jitter of spread σ it is about 1 + σ√(2 ln n). At sixteen thousand devices, a jitter of a couple of per cent costs several per cent of throughput.

Worse is a device that is consistently slow but not broken — a GPU running hot, a degraded link, a failing fan. It raises no error, so failure detection never fires, yet every step waits for it. The Llama 3 team describes exactly this: “even a single straggler can slow down thousands of other GPUs”, and built tools to find the few suspects among sixteen thousand. The workshop simulates both effects.

Below: left, the expected step slowdown against cluster size for the chosen jitter, simulated. Right, for the chosen cluster: an ideal step, the step with jitter, and with one device also consistently slow.

What you will learn
Workshop
Each device’s step time is lognormal with mean 1 and spread σ; the step takes the maximum over n devices, averaged over 100 simulated steps. A single slow device takes 1 + s every step. Throughput loss is 1 − 1/slowdown.

Waiting for the slowest

With 2% jitter per device, eight GPUs lose 2.8% of their throughput to waiting, and 16 384 GPUs lose 7.6% — the expected slowest of sixteen thousand is far out in the tail. Halving the jitter to 1% roughly halves the loss, to 3.9%. Over a run the size of Llama 3.1 405B’s 30.84 million GPU-hours, a 7.6% loss would be about 2.34 million GPU-hours. A single device 20% slow, which raises no error, costs 16.7% — the whole cluster runs at its pace.

The causes are physical and mundane: Meta saw a diurnal 1–2% throughput swing because mid-day heat made GPUs lower their clocks. Finding a straggler means comparing devices: per-rank step and collective timings, flagged when one rank is consistently last, and then draining that host. Real clusters keep a few spare nodes so a suspect can be swapped out without shrinking the job.

Jitter and stragglers are why synchronous training values uniformity over peak speed: a cluster of identical, evenly cooled devices on a balanced network beats a faster but uneven one.

Terms

Straggler
A device that finishes its share of a step late, delaying everyone.
Tail latency
The slowest times in a distribution, which set the pace of synchronous work.
Jitter
Random variation in a device’s step time.
Draining
Removing a suspect host from the job and replacing it with a spare.
A synchronous step waits for the slowest of n devices: jitter costs about σ√(2 ln n), and one slow but working device slows the whole cluster.

Sources

  1. Dean and Barroso, “The Tail at Scale”, CACM 56(2), 2013.
  2. Llama 3 paper, arXiv:2407.21783, section 3.3.4 (stragglers; diurnal 1–2% throughput variation from temperature-driven frequency scaling).