Vol. XV · Running a Training Cluster Plate 8 of 9training report, 6ND, GPU-hours, MFU, consistency check, Llama 3 405B, DeepSeek-V3, attention FLOPsCE

Reading a Training Report.

Parameters, tokens, GPU-hours and peak FLOP/s: four published numbers that must agree, and the utilisation they imply.

Training reports state their numbers separately, but they are not independent. Training compute is about 6ND FLOPs (plate X.1) for N active parameters and D tokens, plus the attention FLOPs that 6ND leaves out. GPU-hours times the chip’s peak is the compute the hardware could have delivered. The ratio is the model FLOPs utilisation the run achieved end to end — including restarts, checkpoint pauses, stragglers, long-context stages and everything else the headline MFU of a tuned step does not include.

The workshop checks two reports. Llama 3: 405 billion parameters, 15.6 trillion tokens, a stated 3.8 × 10²⁵ FLOPs, 30.84 million H100 GPU-hours and a per-configuration MFU of 38–43%. DeepSeek-V3: 37 billion active parameters, 14.8 trillion tokens, 2.664 million H800 GPU-hours for pre-training, 180 000 per trillion tokens, on 2 048 GPUs, trained in FP8.

Below: left, a table of each report’s stated numbers beside the ones recomputed from the others. Right, the implied end-to-end utilisation against the chip’s BF16 and FP8 peaks.

What you will learn
Workshop
C from N and D, optionally with causal attention FLOPs for the stated context length and head dimensions (Llama 3: 8 192 tokens, 128 heads of 128, 126 layers; DeepSeek-V3: 4 096 tokens, 128 heads with 192-dimensional queries and keys and 128-dimensional values, 61 layers). Peaks: 989 TFLOP/s BF16 and 1 979 FP8 for H100 and H800, dense. Llama 3’s GPU-hours are the model card’s total; DeepSeek-V3’s are its pre-training stage.

Four numbers that must agree

For Llama 3, 6ND is 3.79 × 10²⁵ FLOPs, matching the stated 3.8 × 10²⁵; attention adds 4.2% more. Over 30.84 million GPU-hours at 989 TFLOP/s, that is 36.0% of BF16 peak end to end — below the 38–43% of a tuned step, as it should be: the GPU-hours include the effective-time losses of plates XV.1–XV.3 (about 10%), the long-context stage at 38%, annealing and post-training. If the end-to-end figure had come out above the step MFU, something in the report would be wrong.

For DeepSeek-V3, the 180 000 GPU-hours per trillion tokens and the 2.664 million for 14.8 trillion tokens agree (180 thousand recomputed), and 2 048 GPUs make it 54 days — the report’s “less than two months”. Its 37 billion active parameters give 3.29 × 10²⁴ FLOPs by 6ND; attention, at 4 096 tokens with wide heads, adds 14%, more than for Llama because the active model is smaller. The result is 39% of BF16 peak, or 20% of the FP8 peak its matrix multiplies mostly ran at. Neither report states this number; both let you compute it.

The checks here need only arithmetic and a spec sheet. When a report’s numbers do not agree — FLOPs too large for the GPU-hours at any plausible utilisation, say — the usual explanations are a different token count, unstated stages, or a peak figure quoted with sparsity.

Terms

6ND
The training-FLOPs estimate: six FLOPs per active parameter per token.
End-to-end utilisation
Training FLOPs divided by what the GPU-hours could have delivered at peak.
Consistency check
Recomputing one published number from the others.
Active parameters
The parameters used per token, which set the FLOPs of a mixture of experts.
A report’s parameters, tokens, GPU-hours and peak must agree; their ratio is the utilisation the run achieved end to end, always below a tuned step’s.

Sources

  1. Llama 3 paper, arXiv:2407.21783, sections 1 and 3.3; Llama 3.1 model card.
  2. DeepSeek-V3 Technical Report, arXiv:2412.19437, section 1, Table 1 and section 3.3 (FP8 mixed-precision training).