Vol. XXIV · Making Inference Cheap Plate 8 of 10mixture of experts, MoE inference, expert parallelism, DeepSeek-V3, Mixtral, arithmetic intensity, batch sizeCE

Serving a Mixture of Experts.

Cheap to compute per token, expensive to hold — and at scale, expensive to read.

A mixture-of-experts layer (plate VII.4) routes each token to k of E expert feed-forward networks. DeepSeek-V3 has 671 billion parameters but uses 37 billion per token; Mixtral 8×7B has 46.7 billion and uses 12.9. Per token the arithmetic is that of the small active set. But all experts must be in memory, and the bytes a decode step reads depend on how many distinct experts the batch’s tokens choose — for a batch of B tokens routed uniformly, about E(1 − (1 − k/E)^B) of them per layer.

At batch one, a step reads only the active parameters: fast. As the batch grows, the experts touched approach all of them and the step reads nearly the whole model while each expert sees only about B·k/E tokens — so arithmetic intensity grows far more slowly than for a dense model of the active size. The workshop computes, per decode step and assuming uniform routing in BF16, the share of experts touched, the bytes read and the arithmetic intensity.

Below: arithmetic intensity per decode step against batch size for the chosen MoE model and for a dense model of its active size; readouts give experts touched, bytes read and tokens per expert.

What you will learn
Workshop
Parameters from the DeepSeek-V3 and Mixtral reports: total and active counts and routing (DeepSeek-V3: 256 routed experts, 8 per token, plus a shared expert; Mixtral: 8 experts, 2 per token). Expert size e and always-active parameters A derived from those counts; uniform routing; BF16; KV cache ignored. Dense comparison: a dense model with the MoE’s active parameter count.

Huge batches, many GPUs

DeepSeek-V3 must hold 1,342 GB of weights in BF16, yet one token reads only about 74 GB. A batch of 64 tokens already touches most of the routed experts and reads 1,170 GB per step, for an arithmetic intensity of about 4 FLOPs per byte — while each expert sees only 2.0 tokens. To give every expert a batch of 256 tokens per step, the whole batch must hold about 8,192 tokens: MoE models are cheap per token only when served at very large scale.

That is how DeepSeek serves V3 (section 3.4 of its report): decoding units of 320 GPUs with expert parallelism, one routed expert per GPU, attention replicated with data parallelism, and redundant copies of the most-used experts; prefill units of 32 GPUs with 32 redundant experts. Routing is not uniform in practice, so load balancing — duplicating hot experts, placing them to balance traffic — is a serving problem as much as a training one. At small scale, as on a single workstation, an MoE reads nearly all of its large weights every step and loses much of its advantage.

Uniform routing is a simplification: real routers favour some experts, which means some GPUs in an expert-parallel deployment wait for others — hence redundant experts.

Terms

Active parameters
Those used for one token.
Total parameters
All parameters, which must be held in memory.
Expert parallelism
Placing different experts on different GPUs.
Redundant experts
Extra copies of heavily used experts to balance load.
An MoE is cheap in arithmetic, expensive in memory and reads; it pays off only with large batches spread across many GPUs.

Sources

  1. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024, sections 2.1 and 3.4.
  2. Jiang et al., “Mixtral of Experts”, arXiv:2401.04088, 2024.
  3. Rajbhandari et al., “DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training”, ICML 2022.