Vol. VII · Modern Architectures Plate 1 of 10SwiGLU, gated linear unit, GLU, MLP, gating, parameter budgetAL

Gated MLPs.

Let one projection decide, element by element, how much of another gets through.

The MLP of plate III.8 widens a token’s vector, applies a nonlinearity, and projects it back. A gated linear unit (GLU) widens it twice, with two different matrices, passes one copy through a nonlinearity and multiplies the two elementwise: one projection acts as a soft gate on the other. With SiLU as the nonlinearity this is SwiGLU (Shazeer, 2020), now the MLP of Llama, Mistral, Qwen, DeepSeek and most other open models.

A gated MLP has three matrices instead of two. To keep the parameter count equal to a standard MLP with hidden width 4d, the hidden width is cut to about (8/3)d — Llama 3 8B uses 14 336 for d = 4 096, three and a half times d. The workshop compares two recorded runs of micro’s shape that differ only in this choice, with the parameters matched.

Below: left, what each MLP computes for one input direction as its size grows: a GELU unit, and a SwiGLU unit whose gate and value come from different projections. Right, held-out loss of two recorded runs of micro’s shape with a GELU MLP (width 512) and a SwiGLU MLP (width 344), parameters matched.

What you will learn

Write a gated MLP and SwiGLU.
Match a SwiGLU’s parameters to a standard MLP.
Compare gated and ungated MLPs from recorded runs.
Recognise SwiGLU in published configurations.

Workshop

Left: one unit’s output for input t·u along a direction u: GELU(t) for the plain MLP, SiLU(a t)·(b t) for a SwiGLU unit (a = 1, b set by the slider). Right: held-out loss of recorded runs with a GELU MLP (blue) and a SwiGLU MLP (red), same parameter count.

A gate per unit

The two recorded runs use MLPs of almost exactly the same size — 524,288 parameters for GELU at width 512, 528,384 for SwiGLU at width 344 — and identical everything else. After 6 000 steps SwiGLU’s held-out loss is 4.193 nats against GELU’s 4.208, a difference of 0.015. That is the kind of gain Shazeer reported in 2020: small, consistent across tasks, and free in parameters. The left panel shows why gating is more expressive: a SwiGLU unit’s response is a product of two learned projections, so it can be quadratic-like, flat, or switched off depending on what the gate projection sees, where a GELU unit can only bend one input.

Shazeer’s paper offered no mechanism and famously attributed the improvement to “divine benevolence”. What is clear is the engineering: three matrices of width about (8/3)d match the parameters of two of width 4d; Llama 3 8B’s hidden width is 3.5 times d, a little more than 8/3, rounded for hardware efficiency. The cost is one more matrix multiply per MLP, which is why parameter-matched comparisons, not width-matched ones, are the fair test.

Gated units are older than Transformers: the LSTM of 1997 is built from gates, and Yann Dauphin and colleagues introduced gated linear units for convolutional language models in 2016.

Terms

Gated linear unit (GLU): Two projections of the input multiplied elementwise, one through a nonlinearity.
SwiGLU: A GLU with SiLU (swish) as the nonlinearity; the MLP of most open models.
Parameter matching: Shrinking the hidden width so that a three-matrix MLP has the same parameters as a two-matrix one.
Gate: A learned multiplier that controls how much of a signal passes.

SwiGLU gates one projection with another; at about (8/3)d width it costs the same parameters as a 4d GELU MLP and trains slightly better.

Sources

Shazeer, “GLU Variants Improve Transformer”, arXiv:2002.05202, 2020.
Llama 3 8B configuration: arXiv:2407.21783, 2024.