Vol. VII · Modern Architectures Plate 5 of 10load balancing, auxiliary loss, expert collapse, capacity factor, dropped tokens, routerLC

Keeping Experts Busy.

A popular expert gets more practice and becomes more popular — unless the training objective pushes back.

Routing has a rich-get-richer instability. An expert that happens to score well early receives more tokens, so it gets more gradient, improves faster, and scores well even more often; experts that are rarely chosen barely learn. In the worst case the model collapses onto a handful of experts and the rest are dead weight. On hardware there is a second problem: experts usually live on different devices, and if one receives most of the tokens, every other device waits for it.

The standard remedy, from the Switch Transformer, is an auxiliary loss added to the language-model loss: E times the sum over experts of (fraction of tokens sent to the expert) × (average router probability for that expert), which is smallest when both are uniform. Many systems also set a capacity — each expert processes at most a fixed number of tokens per batch — and drop or reroute the overflow. DeepSeek-V3 replaced the auxiliary loss with a bias added to each expert’s score, nudged up or down after each step according to its load.

Below: left, a simulated router: 4 096 tokens sent top-2 to 8 experts, with a skew favouring the first experts, and the tokens dropped by a capacity limit. Right, the share of tokens each expert received in every layer of two recorded runs — with and without the auxiliary loss.

What you will learn
Workshop
Left: a simulated top-2 router over 8 experts for 4 096 tokens; router scores are random plus a skew toward the first experts. Bars: share of assignments per expert (perfect balance is 1/8); dropped share given the capacity factor. Right: share of top-2 assignments per expert (columns) in each layer (rows) of recorded runs of micro’s shape with 8 experts, measured on the last batch.

Pushing back against popularity

In the simulation, a skew of 1 in the router’s scores sends 28.0% of assignments to the busiest expert where balance would give 12.5%, and a capacity factor of 1.25 — each expert takes at most 1.25 times its fair share — drops 14.3% of them. With no skew the busiest expert gets 13.0% and 0.0% are dropped; with skew back at 1 and a capacity factor of 2, 3.0%. A dropped assignment is not lost entirely: the token still passes through the residual stream, and its other expert still runs, but it gets less computation than it should.

The recorded runs show the auxiliary loss doing its job. With it, the shares across all layers and experts stay between 8.1% and 20.0% and the model ends at 4.037 nats. Without it, they range from 2.0% to 36.4%, and the model ends at 4.039 — no worse. At this small scale, with all experts on one device, imbalance costs nothing measurable, but at large scale an imbalance means idle devices and wasted experts, so every production MoE uses some balancing mechanism — the auxiliary loss, DeepSeek-V3’s per-expert bias, or both.

Too strong a balancing loss has its own cost: it forces the router to spread tokens even where one expert really is the best choice. The coefficient α is usually around 0.01, and DeepSeek-V3’s bias method was designed to avoid this trade-off.

Terms

Load balancing
Keeping the number of tokens routed to each expert roughly equal.
Auxiliary loss
An extra term in the training loss that is smallest when routing is uniform.
Capacity factor
The maximum tokens an expert may take, as a multiple of its fair share.
Dropped token
An assignment that overflows an expert’s capacity and is skipped.
Routers drift toward a few favourite experts; an auxiliary loss or a per-expert bias keeps the load spread, and capacity limits decide the overflow.

Sources

  1. Fedus, Zoph and Shazeer, “Switch Transformers”, JMLR 2022, section 2.2 (auxiliary load-balancing loss).
  2. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024, section 2.1.2 (auxiliary-loss-free load balancing).