Vol. XIII · Hardware Plate 5 of 9interconnect, NVLink, NVSwitch, InfiniBand, RoCE, all-reduce, scale-up, scale-out, network bandwidthCE

Interconnect.

A fast fabric inside each box, a slower network between boxes — and an algorithm that must respect both.

In an HGX H100 server, eight GPUs are joined through NVSwitch chips by NVLink at 900 GB/s per GPU in total, 450 GB/s each way — a scale-up domain in which every GPU can reach every other at close to memory speeds. Between servers, each GPU has a network port: Llama 3 was trained over 400 Gb/s RoCE and InfiniBand links, 50 GB/s, about nine times slower than NVLink each way. Newer systems widen the fast domain: a GB200 NVL72 rack joins 72 GPUs in one NVLink fabric.

Collective operations — all-reduce, all-gather, reduce-scatter, all-to-all — move data among a group of devices, and their time is set by the slowest link they cross. A ring all-reduce of S bytes over n devices sends about 2S(n − 1)/n bytes through each link. The workshop times the all-reduce of a model’s gradients inside one server and across servers, which shows why the most communication-hungry parallelism is kept inside the NVLink domain (plate XIV.6).

Below: time to all-reduce the gradients of the chosen model over NVLink inside a server and over a 400 Gb/s network, against the number of GPUs, with the time of the forward and backward compute for comparison.

What you will learn
Workshop
Gradient all-reduce for the chosen model (BF16 gradients, 2 bytes per parameter) with a ring algorithm, link bandwidth 450 GB/s each way for NVLink within a server and 50 GB/s for a 400 Gb/s network port, 5 µs per step. The compute time is a step of 1 million tokens per GPU group at 40% of H100 peak, 6N FLOPs per token, spread over the GPUs.

Fast inside, slower between

Llama 3 8B’s BF16 gradients are 16.1 GB. All-reducing them across the eight GPUs of one server over NVLink takes 6.25 × 10⁻² seconds; over a 400 Gb/s network port — 50 GB/s, 9 times less bandwidth each way — 5.62 × 10⁻¹. A ring all-reduce costs about 2S ÷ B however many devices join, so at 1 024 GPUs the network all-reduce still takes 6.52 × 10⁻¹ seconds while each GPU’s share of a million-token step’s compute has fallen to 1.19 × 10⁻¹: communication would dominate if it were not hidden.

Two consequences follow. Communication must overlap with computation — gradients are all-reduced layer by layer while the backward pass continues — and only data parallelism, which communicates once per step, is spread across the slow network. Tensor parallelism, which communicates inside every layer, stays within the NVLink domain; Llama 3 used tensor parallelism of 8, exactly one server (plate XIV.10). Wider fast domains such as the 72-GPU NVL72 relax that constraint, which is a large part of their purpose.

Network bandwidth figures are quoted in bits per second, NVLink figures in bytes, and NVLink figures often as the sum of both directions. Converting to the same unit and direction before comparing avoids errors of eight and of two.

Terms

Scale-up domain
A group of accelerators joined by the fastest link, such as NVLink within a server or rack.
Scale-out network
The network between servers: InfiniBand or Ethernet (RoCE).
All-reduce
A collective in which every device ends with the sum of all devices’ data.
Ring algorithm
An all-reduce that passes chunks around a ring, using every link fully.
Interconnect has a fast domain inside each server or rack and a slower network between them; parallelism is placed so that the chattiest traffic stays on the fast links.

Sources

  1. NVIDIA H100 datasheet (NVLink 900 GB/s); NVIDIA GB200 NVL72 specifications.
  2. Llama Team, AI @ Meta, arXiv:2407.21783, 2024, section 3.3.1 (400 Gbps RoCE and InfiniBand).
  3. Patarasuk and Yuan, “Bandwidth optimal all-reduce algorithms for clusters of workstations”, JPDC 69(2), 2009.