Vol. V · The Transformer Plate 6 of 10FLOPs, 2N, 6N, forward pass, backward pass, training compute, attention FLOPsC

FLOPs per Token.

Two FLOPs per parameter to read a token, six to learn from it — and the arithmetic of every training run follows.

Every weight in a matrix multiply is used once per token: one multiplication and one addition, two floating-point operations. So a forward pass through a model with N parameters (not counting the embedding lookup, which is a copy, not a multiply) costs about 2N FLOPs per token. The backward pass computes two gradients for every forward multiply — one for the input, one for the weight — so it costs about twice the forward: 4N. Training therefore costs about 6N FLOPs per token, and a run on D tokens about 6ND.

Two terms sit outside the rule. The output logits cost 2Vd per token whether or not the embedding is tied, and attention adds 4Td per layer, which matters only at long contexts (plate IV.7). The workshop computes all three exactly for real configurations and compares the total with 2N.

Below: the forward FLOPs per token of the chosen configuration, split into the matrix multiplies of the blocks, the output logits and attention, at the chosen context length. The readings give training FLOPs per token and, for published runs, the total training compute.

What you will learn

Derive 2N FLOPs per token for a forward pass.
Derive 6N FLOPs per token for training.
Add the logit and attention terms.
Estimate a training run’s total compute from N and D.

Workshop

Forward FLOPs per token of the chosen configuration at the chosen context: block matrix multiplies (2 × non-embedding parameters), output logits and attention.

Six N D

For GPT-3 at its 2 048-token context, a forward token costs 3.59 × 10¹¹ FLOPs, 1.028 times 2N — the logits and attention add a few per cent. Training costs three times that per token. Multiply 6N by the 300 billion tokens GPT-3 was trained on and the run comes to 3.14 × 10²³ FLOPs; the paper reports 3.14 × 10²³, a difference of 0.1%. The same arithmetic for the published Llama 3 8B — 8 billion parameters, about 15 trillion tokens — gives 7.23 × 10²³ FLOPs, more than twice GPT-3’s budget for a model twenty times smaller: a deliberate choice examined in plate X.5.

At 8 192 tokens of context the Llama 3 8B forward token is 1.202 × 2N, and attention is 22% of it; at 131 072 tokens attention is 82%, and the 6N rule badly undercounts. These are the numbers a lab uses to plan: compute budget = 6 × parameters × tokens, divided by what the hardware sustains (Vol. XIII), gives the length of a run. They are also the course’s first warning about approximations: 6N ignores attention, recomputation (plate XIV.2) and the logits, and every plate that uses it says so.

The 6N rule appears in Kaplan et al. (2020); Hoffmann et al. (2022) computed FLOPs more carefully, including attention and logits, and found the difference small at the context lengths they used.

Terms

FLOP: One floating-point operation: an addition or a multiplication.
2N rule: A forward pass costs about two FLOPs per parameter per token.
6N rule: Training costs about six FLOPs per parameter per token: forward plus a backward twice as expensive.
Training compute: Total FLOPs of a run, about 6 × parameters × tokens.

A forward token costs 2N FLOPs, a training token 6N, and a run 6ND — plus logits, and attention at long contexts.

Sources

GPT-3: Brown et al., NeurIPS 2020, Table D.1 (3.14 × 10²³ training FLOPs).
Llama 3: Llama Team, AI @ Meta, arXiv:2407.21783, 2024 (about 15 trillion pretraining tokens).