Vol. XIV · Parallelism Plate 9 of 11expert parallelism, mixture of experts, all-to-all, dispatch, combine, node-limited routing, DeepSeek-V3CE

Spreading the Experts.

Put different experts on different devices and send each token to where its experts live — then bring it back.

A mixture-of-experts layer (plate VIII.5) holds many expert MLPs but each token uses only a few. Expert parallelism places the experts on different devices: with 256 experts over 64 devices, each holds 4. A token’s hidden vector is dispatched to the devices holding its chosen experts with an all-to-all, processed there, and the results are combined back with a second all-to-all. Expert parameters per device fall by the expert-parallel degree, and each device’s experts see a large batch of tokens gathered from all the others, which keeps their matrix multiplies efficient.

The price is traffic that grows with the tokens and with how many experts each token uses. DeepSeek-V3 routes each token to 8 of 256 experts; its report notes that across nodes the computation-to-communication ratio was about 1:1. The workshop estimates that ratio for one MoE layer of DeepSeek-V3’s shape and shows the three levers its authors pulled: restricting each token to at most 4 nodes, dispatching in FP8, and overlapping the exchange with other work.

Below: left, all-to-all time relative to expert compute for one MoE layer, against the expert-parallel degree, for DeepSeek-V3’s shape. Right, expert parameters per device.

What you will learn
Workshop
One MoE layer of DeepSeek-V3’s shape (d = 7 168, expert hidden size 2 048, 8 of 256 routed experts plus 1 shared), forward and backward, per token. Compute at 45% of an H800’s BF16 peak (the H100’s arithmetic). Traffic: dispatch and combine in each direction to the expected number of distinct remote destinations — GPUs over NVLink (160 GB/s, the report’s figure for H800) inside a node of 8, nodes over a 400 Gb/s network (50 GB/s) beyond, then forwarded over NVLink inside the destination node, the slower leg setting the time — with combine in BF16 and dispatch in BF16 or FP8.

Traffic that rivals compute

With 64-way expert parallelism over 8 nodes, a token’s 8 experts land on 4.59 remote nodes on average, and dispatching and combining it moves 263.5 KB per layer — 99% of the time its experts take to compute. That is the report’s “approximately 1:1”. Limiting each token to 4 nodes cuts it to 75%; adding FP8 dispatch, to 56%. Inside one node, over NVLink, 8-way expert parallelism costs only 47%.

Even at its reduced level, the exchange is too slow to sit on the critical path, so DeepSeek-V3’s DualPipe schedule runs one micro-batch’s all-to-all while another computes, and dedicates a few of each GPU’s streaming multiprocessors to driving the communication. The design follows the same rule as tensor parallelism: the heavy traffic goes where the links are fastest, and what must cross the network is limited, narrowed and hidden.

Expert parallelism interacts with load balancing (plate VIII.6): if one expert is popular, the device holding it receives more tokens and every other device waits — so balancing the router is a systems requirement as much as a modelling one.

Terms

Expert parallelism
Placing different experts on different devices.
Dispatch
The all-to-all sending each token to the devices of its chosen experts.
Combine
The all-to-all returning expert outputs to the token’s home device.
Node-limited routing
Restricting the experts a token may use to a few nodes, capping network traffic.
Expert parallelism divides the experts but adds two all-to-alls per layer each way; across nodes they rival compute until routing is limited, formats narrowed and the exchange overlapped.

Sources

  1. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024, sections 3.2.1–3.2.2 (cross-node expert parallelism with a computation-to-communication ratio of about 1:1; each token dispatched to at most 4 nodes; FP8 dispatch; DualPipe overlap).
  2. Lepikhin et al., “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding”, ICLR 2021.