Vol. V · The Transformer Plate 5 of 10parameter count, 12 d squared, embedding parameters, configuration, model sizeAC

Counting Parameters.

Width, depth, vocabulary: three numbers, and the size of a model to within a fraction of a per cent.

A standard block has four d × d attention matrices (queries, keys, values, output) and an MLP with two d × 4d matrices: 4d² + 8d² = 12d² parameters, plus a few thousand in its norms. A model of L blocks therefore has about 12 L d² parameters in its blocks, which is usually called its non-embedding size, plus V × d for the embedding table (twice that if the output head is untied) and, for learned positions, T × d. Grouped-query attention and gated MLPs change the 12 slightly; mixtures of experts change the MLP term a great deal (Vol. VII).

This arithmetic is how a model is planned, and the workshop holds it to the published totals of real models. It also shows how differently sizes are spent: in a tiny model like micro the embedding is the largest part; in GPT-3 the blocks are over 99% of the total.

Below: the parameters of the chosen configuration split by part, computed from its published shape by the course’s compute engine, beside the published total (for micro, the count of its actual weights).

What you will learn
Workshop
Parameters of the chosen configuration by part: embedding (and untied head), positions, attention, MLP and norms, from its published shape. The readings compare the total with the published figure.

Twelve d squared per block

From its published shape — 96 layers of width 12 288 — GPT-3 comes to 175 billion parameters, against the published 175 billion, a difference of −0.23%; the 12 L d² rule alone gives 174 billion, because the embedding is only 0.4% of this model. GPT-2 small comes to 124 million against a published 124 million (the small remainder is biases, which the count leaves out), with its embedding 31% of the total. The published Llama 3 8B shape — grouped-query attention, a gated MLP of width 14 336, an untied vocabulary of 128 256 — comes to 8.03 billion, matching the published 8.03 billion to 0.00%.

micro is the opposite extreme: 1.85 million parameters, the same as the count of its stored weights, of which the embedding is 57%. At that size most of the model is vocabulary, which is why the volumes that train small models use small vocabularies or report non-embedding parameters. The scaling laws of Vol. X are written in non-embedding parameters for the same reason: the embedding contributes little computation per token, and counting it would make small models look larger than they are.

The 12 L d² approximation appears in Kaplan et al.’s 2020 scaling-law paper, which counts N = 2 d L (2 d_attn + d_ff) and excludes embeddings — the convention most later work follows.

Terms

Non-embedding parameters
Parameters in the blocks, excluding the embedding and output tables; about 12 L d².
Configuration
The handful of numbers that fix a model’s shape: d, L, H, KV, ff, V, T.
Untied head
A separate output matrix rather than reusing the embedding; adds V d parameters.
Grouped-query attention
Fewer key and value heads than query heads; shrinks attention’s parameter count slightly.
A standard block has 12 d² parameters; a model has about 12 L d² plus V d for its vocabulary — and published totals check it.

Sources

  1. GPT-2 small: Radford et al., 2019.
  2. GPT-3: Brown et al., NeurIPS 2020, Table 2.1.
  3. Llama 3 8B: Llama Team, AI @ Meta, arXiv:2407.21783, 2024, Table 3.