Vol. XV · Running a Training Cluster Plate 2 of 9checkpointing, Young–Daly interval, effective training time, goodput, restart, storage bandwidth, in-memory replicasCE

Checkpointing.

Save too often and you spend the run writing; too rarely and you spend it redoing. The square root in between.

A checkpoint saves everything needed to resume: weights, optimiser state, the data loader’s position and the random number generators. When the job fails, it restarts from the last checkpoint, losing on average half an interval of work, plus the time to restart. Writing the checkpoint pauses training for δ, so with an interval τ the share of time lost is about δ/τ + (τ/2 + r)/M, where M is the job’s failure interval and r the restart time. Young (1974) and Daly (2006) showed it is smallest at τ* ≈ √(2δM).

For a large model the checkpoint is big — the 405-billion-parameter model’s states are 6.5 TB — but it is written in parallel by every GPU to a distributed file system; Llama 3’s offered 2 TB/s sustained. The workshop finds the best interval for a model, a storage bandwidth, a failure interval and a restart time, and shows which term dominates the loss.

Below: the share of training time lost against the checkpoint interval, split into writing, redone work and restarts, with Young and Daly’s optimum marked.

What you will learn
Workshop
Checkpoint size: 16 bytes per parameter (BF16 weights, FP32 master weights and two Adam moments, and BF16 gradients are not needed but are counted as a margin), written at the chosen aggregate storage bandwidth with training paused. The share of time lost is δ/τ for writing, τ/(2M) for redone work and r/M for restarts.

The restart is the cost

With Llama 3 405B’s 6.5 TB of states written at 2 TB/s, a checkpoint takes 3.3 seconds, and with a failure every 3.1 hours the best interval is every 4.5 minutes. At that interval 7.8% of the time is lost and 92.2% is effective — close to the “higher than 90%” Meta reports. But 69% of the loss is the ten-minute restart, which no choice of interval can touch. Cut the restart to two minutes and the effective time rises to 96.5%; at a hundredth of the storage bandwidth it falls to 60.5%.

That is why the large labs attack restart time. Gemini’s team kept redundant copies of the model state in memory and recovered from an intact replica instead of from persistent storage, which raised the goodput of their largest job from 85% to 97%. Asynchronous checkpointing — copying the state to host memory in a moment and writing it to storage in the background — attacks δ the same way. And a cluster that failed only once a day would lose 1.6% at its own optimum.

Goodput is the share of the cluster’s time spent on progress that is kept: it excludes the time redone after failures, the time spent restarting and checkpointing, and steps later discarded.

Terms

Checkpoint
A saved copy of everything needed to resume training.
Young–Daly interval
The checkpoint interval √(2δM) that minimises lost time.
Effective training time
The share of elapsed time spent on useful training.
Goodput
Useful progress per unit of cluster time.
Checkpoint every √(2δM); at scale the restart, not the write, is the largest loss, so the best systems restart from memory.

Sources

  1. Young, “A first order approximation to the optimum checkpoint interval”, CACM 17(9), 1974.
  2. Daly, “A higher order estimate of the optimum checkpoint interval for restart dumps”, FGCS 22(3), 2006.
  3. Llama 3 paper, arXiv:2407.21783, section 3.3 (storage fabric; above 90% effective training time).
  4. Gemini Team, arXiv:2312.11805, 2023, section 3 (goodput raised from 85% to 97% with in-memory replicas).