Vol. VII · Modern Architectures Plate 6 of 10Mixtral, DeepSeek-V3, active parameters, memory, FLOPs per token, fine-grained experts, shared expertsCE

Total and Active Parameters.

Pay for memory by the total, pay for arithmetic by the active — and decide how finely to slice the experts.

Two published models show the range. Mixtral 8x7B (2024) has 8 experts in each of 32 layers and routes each token to 2: about 47 billion parameters in total, about 13 billion active. DeepSeek-V3 (December 2024) goes much further: 256 small routed experts plus one shared expert that every token uses, with 8 routed experts active — 671 billion parameters in total, 37 billion active per token.

The economics follow from two rules. Memory for weights scales with the total: every expert must be held somewhere, at 2 bytes per parameter in 16-bit precision. Compute per token scales with the active: about 2 FLOPs per active parameter. Many small experts instead of a few large ones — fine-grained experts — give the router vastly more combinations to choose from at the same active size, and a shared expert holds what every token needs so the routed ones can specialise.

Below: left, total and active parameters of published models, dense and MoE, with the dense model of the same compute. Right, for the selected model, the memory its weights need in 16-bit precision, the forward FLOPs per token, and the number of different expert combinations a token can be routed to.

What you will learn
Workshop
Left: total (blue) and active (red) parameters of Llama 3 8B (dense), Mixtral 8x7B and DeepSeek-V3, from their reports. Readings for the selected model: weight memory at 2 bytes per parameter, forward FLOPs per token at 2 per active parameter, and the number of ways to choose its routed experts in one layer.

Memory by the total, arithmetic by the active

Mixtral holds 3.6 times the parameters it uses per token. Its weights need about 93.4 GB in 16-bit precision — 2 accelerators of 80 GB — while each token costs about 2.58 × 10¹⁰ forward FLOPs, roughly what a dense 13-billion-parameter model would cost. DeepSeek-V3 pushes the ratio to 18.1: about 1.3 TB of weights, but per-token compute like a dense model of 37 billion parameters. Its 256 routed experts, choosing 8, can be combined in about 10^14.6 ways per layer, against Mixtral’s 28.

When does sparsity pay? When compute is the binding cost — training, and serving at large batch where weights are loaded once and shared by many tokens — an MoE delivers more quality per FLOP. When memory or memory bandwidth binds — a single user on one device, where every expert must be resident and loaded (plate XXIV.8) — an MoE costs like the large model it is. That is why MoE is the norm for frontier systems served at scale, and why the smallest open models, meant for one device, are usually dense.

The name “8x7B” is misleading: the experts replace only the MLPs, so attention and embeddings are shared and the total is about 47 billion, not 56.

Terms

Shared expert
An expert that every token uses, alongside the routed ones.
Fine-grained experts
Many small experts instead of a few large ones, giving the router more combinations.
Weight memory
Bytes needed to hold the parameters: total parameters × bytes per parameter.
FLOPs per token
Arithmetic for one token’s forward pass: about 2 × active parameters.
An MoE costs memory by its total parameters and compute by its active ones; it pays off where compute, not memory, is the binding cost.

Sources

  1. Jiang et al., “Mixtral of Experts”, arXiv:2401.04088, 2024.
  2. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024, section 2.1.
  3. Dai et al., “DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models”, arXiv:2401.06066, 2024 (fine-grained and shared experts).