Vol. VII · Modern Architectures Plate 4 of 10mixture of experts, MoE, router, top-k routing, sparse model, active parameters, total parametersAC

Mixture of Experts.

Many MLPs per layer, a router that picks two of them for each token — more parameters without more arithmetic.

In a dense Transformer every token passes through every parameter. A mixture-of-experts (MoE) layer keeps attention as it is but replaces the MLP with E separate MLPs, the experts, and a router: a small linear layer that scores the experts for each token. Only the top K experts by score run, and their outputs are added, weighted by the router’s normalised scores. Since K is much smaller than E, the model holds E MLPs’ worth of parameters but spends only K MLPs’ worth of arithmetic on each token.

This splits one number into two. Total parameters decide how much the model can store, and what memory it needs; active parameters decide the compute per token. The workshop counts both for a micro-sized model with experts, and compares three recorded runs trained on the same tokens: a dense model with the active compute of an 8-expert, top-2 MoE; the MoE itself; and a dense model as wide as all eight experts put together.

Below: left, total and active parameters of a micro-sized model as the number of experts E and the number chosen K change. Right, held-out loss of three recorded runs: dense with the MoE’s active size, the 8-expert top-2 MoE, and dense with the MoE’s total size.

What you will learn

Describe a mixture-of-experts layer and its router.
Separate total parameters from active parameters.
Compare an MoE with dense models of equal compute and equal size.
Explain why MoE has become common at the frontier.

Workshop

Left: parameters of a model with micro’s attention (d = 128, 4 layers) and E experts of width 512, K active, split into total and active. Right: held-out loss of recorded runs, all 6 000 steps on the same 49 million tokens: dense with MLP width 1 024 (equal active compute to the MoE), the MoE with 8 experts of width 512 and top-2 routing, and dense with width 4 096 (equal total MLP parameters).

Parameters you hold, parameters you use

At the slider’s starting point — 8 experts, 2 per token — the model holds 4.46 million non-embedding parameters but uses 1.32 million on each token: it holds 3.39 times what it uses. With 64 experts and still 2 per token it would hold 25.2 times what it uses. The recorded runs put this to the test on identical data. The MoE (4.46 million non-embedding parameters, about 1.31 million of them active) ends at 4.037 nats; the dense model with the same compute per token ends at 4.166, 0.129 nats worse. More surprising, the dense model as large as all the experts together (4.46 million parameters, four times the MoE’s compute per token) ends at 4.074 — slightly worse than the MoE. One run per arm at this scale cannot say the MoE is better than an equally large dense model in general; published comparisons usually find the opposite at equal total size. What it does show is how much of the dense model’s benefit the MoE keeps at a quarter of the arithmetic.

That is the case for MoE in one picture: for the same arithmetic per token, more parameters to store what has been learned. It does not come free. The MoE needs memory for every expert, its experts must be spread across devices in large models (plate XIV.9), and its routing must be kept balanced (the next plate). Mixtral 8x7B, DeepSeek-V3, Qwen3’s largest models and, by public reports, several closed frontier models are mixtures of experts.

The router is just a linear layer followed by a softmax over experts. Nothing tells an expert what to specialise in; studies of trained MoE models find experts that prefer certain token types or syntax far more often than recognisable topics.

Terms

Mixture of experts (MoE): A layer with many MLPs, the experts, of which a router activates only a few per token.
Router: A small layer that scores the experts for each token; the top K are used.
Top-K routing: Sending each token to the K highest-scoring experts and weighting their outputs.
Total vs active parameters: All parameters the model holds, against those used for one token.

An MoE layer holds many experts but runs only K per token, so it buys parameters without buying compute.

Sources

Shazeer et al., “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR 2017.
Fedus, Zoph and Shazeer, “Switch Transformers”, JMLR 2022.
Jiang et al., “Mixtral of Experts”, arXiv:2401.04088, 2024.