Vol. XV · Running a Training Cluster Plate 6 of 9observability, monitoring, alerts, training loss, held-out loss, gradient norm, throughput, incidentsLE

Watching a Run.

Four curves, four alarms, and an incident that shows up in only one of them.

A large run is monitored like any production system, with dashboards and alerts on a few metrics. The training loss and the gradient norm come from every step; a held-out loss is evaluated every so often on data the model never trains on; throughput — tokens per second per GPU — measures the machinery rather than the model. Each kind of incident leaves a different signature across them, and which alert fires first decides how quickly it is found.

The workshop trains a real model — softmax regression, ten classes, fresh data every step — for 600 steps, and at step 300 injects one incident: a five-step learning-rate spike, a corrupt data shard with shuffled labels, the same batch fed sixty times, or a node that becomes 30% slower. Four alerts watch: a loss jump, a gradient-norm jump, a gap between held-out and training loss, and a throughput drop.

Below: training loss (thin) and held-out loss (dots), gradient norm, and throughput over the run, with the incident marked. The readouts give the step at which each alert first fired.

What you will learn

Name the core metrics of a training run.
Match each incident to its signature.
Explain why repeated data hides in the training loss.
Design alerts that catch different failures.

Workshop

Softmax regression on 20 features, 10 classes, batches of 64 drawn fresh each step, SGD at 0.1, 600 steps; held-out loss on 400 fixed examples every 10 steps. Incidents from step 300: learning rate × 150 for 5 steps; labels shuffled for 30 steps; one batch repeated for 60 steps; throughput 30% lower. Throughput has a 2% lognormal jitter. Alerts as in the formula, with the gradient-norm alert at 3 × its running mean.

Every incident has a signature

With one batch repeated sixty times, the training loss looks better than ever — the model is memorising that batch — so the loss alert stays quiet until the repetition ends and fresh data arrives, at step 362. Only the gap between held-out and training loss gives it away, at step 339. A corrupt shard fires the loss and gradient-norm alerts together at step 300; a learning-rate spike fires the loss alert at step 302; a slow node changes nothing about the model and is caught only by throughput, at step 297. With no incident, 0 of the four alerts fire.

Real dashboards add more — loss per data source, the largest activations and attention logits (plate XI.7), per-rank step times (plate XV.3), memory, temperatures and network counters — but the lesson is the same: watch the model and the machine separately, keep a held-out loss so that memorisation cannot hide, and set thresholds that a healthy run does not trip. Published logbooks such as OPT-175B’s show what happens without enough of this: dozens of manual restarts and learning-rate changes after divergences found late.

A sudden improvement deserves as much suspicion as a sudden degradation: a training loss that falls faster than it should often means the model is seeing data it has seen before, or the evaluation data has leaked into training (plate X.4).

Terms

Held-out loss: Loss on data excluded from training, evaluated periodically.
Gradient norm: The size of the gradient, a sensitive signal of instability.
Throughput: Tokens processed per second, a measure of the machinery.
Alert: An automatic notification when a metric crosses a threshold.

Watch the model and the machine separately: each incident has its own signature, and a held-out loss is what exposes memorisation.

Sources

Llama 3 paper, arXiv:2407.21783, section 3.3.4 (NCCL flight recorder; tracing collectives to find hangs and stragglers).
OPT-175B logbook, Zhang et al., arXiv:2205.01068, 2022 (restarts after loss divergences).