Vol. XIII · Hardware Plate 6 of 9MFU, HFU, model FLOPs utilisation, efficiency, throughput, tokens per second, Llama 3, PaLMCE

Model FLOPs Utilisation.

Count the FLOPs the model needs, divide by the time taken and the chips’ peak — and see how much of the hardware the run really used.

Model FLOPs utilisation (MFU) compares what a run achieves with what its hardware could do: the model’s training FLOPs per token — about 6N, plus attention — times tokens processed per second, divided by the total peak FLOP/s of the chips. It counts only the arithmetic the model mathematically requires, so work done twice, such as recomputing activations to save memory (plate XIV.2), does not inflate it; hardware FLOPs utilisation (HFU) counts that extra work too.

Published numbers are a useful anchor. PaLM’s 540-billion-parameter model ran at 46.2% MFU on TPU v4. Llama 3 405B, on 8 192 to 16 384 H100s, ran at 38–43% MFU in BF16 depending on the parallel layout and sequence length. The workshop turns a measured throughput into MFU, and back.

Below: left, MFU for the chosen model size and throughput per GPU on an H100, against the throughput. Right, Llama 3’s published configurations and their MFU.

What you will learn
Workshop
Left: MFU of one H100 (989 dense BF16 TFLOP/s) for the chosen model size and sequence length, against tokens per second per GPU; the attention term 12 L d T uses the Llama 3 shape scaled to the size. Right: Llama 3 405B’s reported configurations (arXiv:2407.21783, Table 4): GPUs, sequence length and MFU.

Where the other 60% goes

A 405-billion-parameter model needs about 2.63 × 10¹² training FLOPs per token at sequence length 8 192, 7.7% of them in attention. At 40% MFU an H100 therefore processes only about 150 tokens per second for it; at 200 tokens per second it runs at 53.2%. For Llama 3 8B the same 40% means about 6,478 tokens per second per GPU. These numbers make claims checkable: a throughput figure and a model size give the MFU, and an implausibly high MFU usually means a mistake in the FLOP count.

Llama 3 reports 38–43% and PaLM 46.2% (57.8% counting recomputation). The rest is lost in identifiable places: memory-bound operations that cannot use the tensor cores (plate XIII.3), communication not hidden behind compute (plate XIII.5), pipeline bubbles (plate XIV.7), stragglers and restarts (Vol. XV), and the gap between the peak a chip can sustain on a perfect matrix multiply and what real kernels achieve. Most of Part Three is about recovering that utilisation.

MFU uses the model’s required FLOPs, so it is comparable across codebases; HFU depends on how much a codebase recomputes. A run that recomputes all activations can show high HFU and lower MFU for the same speed.

Terms

MFU
Model FLOPs utilisation: required model FLOP/s divided by peak.
HFU
Hardware FLOPs utilisation: all FLOP/s actually executed, including recomputation, divided by peak.
Throughput
Tokens processed per second, per GPU or in total.
Rematerialisation
Recomputing activations in the backward pass instead of storing them.
MFU — required FLOPs per second over peak — runs near 40% for large models; the rest is lost to memory-bound work, communication, bubbles and stalls.

Sources

  1. Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways”, JMLR 2023, section 4.
  2. Llama Team, AI @ Meta, arXiv:2407.21783, 2024, section 3.3 and Table 4.
  3. Korthikanti et al., “Reducing Activation Recomputation in Large Transformer Models”, MLSys 2023 (MFU and HFU).