Silent Data Corruption.
A bit flips, nothing crashes, and the run carries on with a wrong number. Which flips matter, and what catches them?
A defective chip can compute a wrong result without any error being raised: silent data corruption. Studies at Meta and Google found a small fraction of processors that miscompute occasionally under particular conditions. Llama 3’s snapshot attributed 6 interruptions to it in 54 days, and the Gemini report expected it to affect training “every week or two” at its scale. A training run is especially exposed because a corrupted value is not an isolated wrong answer: it enters the weights and every later step builds on it.
The workshop trains a real model — least squares with 32 parameters — on two data-parallel replicas that average their gradients each step, and at step 150 flips one bit of a 32-bit float in replica B: either in its gradient before the average or in its weights after the update. Three detectors watch: a loss-spike alarm, a gradient-norm alarm and a cross-replica checksum that compares the replicas’ weights.
Below: left, held-out loss of the corrupted run and an identical clean run. Right, the largest difference between the two replicas’ weights. Readouts say which detector fired first and how much the final loss was harmed.
- Explain silent data corruption.
- Relate the harm of a bit flip to the bit’s position.
- Explain why corruption before an all-reduce leaves replicas agreeing.
- Compare loss, gradient-norm and checksum detectors.
Loud, harmless, or silent
Flipping the top exponent bit of a gradient multiplies it by about 2¹²⁸: the loss alarm fires at step 151 and the run is ruined, the final loss 6.8 × 10⁶⁴ times the clean one. Flipping exponent bit 26 changes the gradient by a factor of 2¹⁶ or so; the gradient-norm alarm fires, but plain SGD recovers and the final loss is 1.000 times the clean one. A mantissa flip — bit 3 — changes nothing measurable (1.000) and nothing fires. In none of these does the checksum fire, because the corruption entered before the all-reduce and both replicas applied the same wrong update.
Corrupt a weight after the update instead, and the replicas disagree from that step on. A low mantissa flip in a weight is invisible in the loss (1.000) but the checksum sees it at once; flipping bit 25 leaves the final loss 32.8 times the clean one — the replicas stay apart by the flip, since both apply the same averaged updates — and every detector fires. Checking only every 100 steps it is found at step 200. This is why Gemini’s team relied on deterministic replay: when a symptom appears, recomputing the suspect steps on other hardware shows which machine computed differently, even when the replicas agreed.
Terms
- Silent data corruption
- A wrong result computed without any error being raised.
- Bit flip
- One bit of a stored or computed value inverted by a fault.
- Cross-replica checksum
- Comparing a hash of the weights across data-parallel replicas, which should be identical.
- Deterministic replay
- Recomputing steps exactly, on other hardware, to find which machine erred.
Sources
- Dixit et al., “Silent Data Corruptions at Scale”, arXiv:2102.11245, 2021.
- Hochschild et al., “Cores that don’t count”, HotOS 2021.
- Llama 3 paper, arXiv:2407.21783, Table 5.
- Gemini Team, arXiv:2312.11805, 2023, section 3 (SDC every week or two; deterministic replay; proactive scanners).