Vol. VII · Modern Architectures Plate 9 of 10model shape, width, depth, aspect ratio, parameter budget, Kaplan, published configurationsAL

Wide or Deep.

Spend the same layer parameters on fewer, wider layers or more, narrower ones — and check what the count leaves out.

A Transformer’s non-embedding parameters are about 12 L d²: L layers, each with 4 d² in attention and 8 d² in the MLP. The same budget can buy one layer of width 256 or sixteen of width 64. Wider layers give each token a richer vector; deeper stacks give more steps of computation in sequence, and more attention layers in which information can be moved and combined (the induction heads of plate IV.6 need at least two).

Kaplan and colleagues (2020) found that at fixed non-embedding parameters the loss is remarkably insensitive to shape across a wide range of aspect ratios, worsening by only a few per cent between ratios of about 10 and about 200. They excluded the embedding on purpose: in large models it is a small share of the parameters. The recorded runs repeat the test at the course’s scale, where it is not small: five models of about 0.8 million non-embedding parameters, from 1 layer of width 256 to 16 layers of width 64, on the same data for the same steps.

Below: left, final held-out loss of five recorded runs of equal non-embedding size against their depth. Right, each model’s parameters split into the layers and the embedding, which grows with width.

What you will learn
Workshop
Left: final held-out loss after 6 000 steps of recorded runs with about 0.8 million non-embedding parameters: 1 × 256, 2 × 192, 4 × 128, 8 × 96, 16 × 64 (layers × width). Right: each model’s parameters, split into the layers (blue) and the tied embedding and position table (grey), in millions.

What the count leaves out

With the layers’ parameters held near 0.8 million, the wide shapes won: 2 layers of width 192 ended at 4.098 nats and 1 layer of width 256 at 4.118, against 4.208 for micro’s 4 × 128, 4.280 for 8 × 96 and 4.496 for 16 layers of width 64. That is not Kaplan’s flat curve, and the reason is in the right panel. The embedding — 8 192 rows of width d, shared with the output layer — grows with width: it is 73% of the 1 × 256 model’s 2.92 million parameters and 40% of the 16 × 64 model’s 1.32 million. At this scale the comparison is really between models of very different total size, and the larger tables win.

Depth still shows its value at the edge: the single-layer model, with the largest embedding of all, finished behind the two-layer one — plausibly because one layer cannot compose attention (the induction heads of plate IV.6 need two). In large models the embedding is a small share — about 0.4% of GPT-3’s parameters — and the shape question becomes Kaplan’s: a broad, flat optimum. Published models settle on widths about 60 to 130 times their depth: GPT-3 at 128, Llama 3 8B at 128. The general lesson is about counting: before comparing two models “of the same size”, check which parameters the size leaves out.

Depth has a cost the parameter count hides: layers run one after another, so a deeper model has higher latency per token on the same hardware, while width parallelises well. Published models settle on widths roughly 60 to 130 times their depth.

Terms

Width
The model dimension d: the length of each token’s vector.
Depth
The number of layers L.
Aspect ratio
Width divided by depth, d / L.
Non-embedding parameters
Parameters outside the embedding and output layers, about 12 L d².
In large models shape matters little across a broad range; in small ones, check what the parameter count leaves out — here the embedding, which grows with width.

Sources

  1. Kaplan et al., “Scaling Laws for Neural Language Models”, arXiv:2001.08361, 2020, section 3.1 and Figure 5.
  2. Configurations: GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), Llama 3 8B (arXiv:2407.21783), DeepSeek-V3 (arXiv:2412.19437).