Vol. XIV · Parallelism Plate 7 of 11pipeline parallelism, stages, micro-batches, bubble, GPipe, 1F1B, interleaved schedule, point-to-pointCE

Pipelines and Bubbles.

Give each device a block of layers and pass the activations along. The cost is idle time at the ends of every step.

In pipeline parallelism the model is cut into p stages of consecutive layers, one per device or group. Stage 1 runs the forward pass of a micro-batch and sends its activations to stage 2, and so on; the backward pass flows the other way. Only one activation tensor per micro-batch crosses each boundary, so pipelines tolerate slow links — they are how large models span many servers. The catch is the bubble: at the start of each step later stages wait for the first micro-batch to arrive, and at the end earlier stages wait for the last backward to return.

Splitting the batch into m micro-batches fills the pipe. GPipe (Huang et al., 2019) runs all forwards then all backwards; its idle fraction is (p − 1)/(m + p − 1), and it must keep the activations of all m micro-batches. The 1F1B schedule of PipeDream (Narayanan et al., 2019) has the same bubble but alternates one forward with one backward once the pipe is full, so each stage holds at most p micro-batches’ activations. The workshop draws both schedules.

Below: the schedule of p stages processing m micro-batches — forward passes in blue, backward passes (twice as long) in green, idle time blank — with the bubble fraction and the activations each schedule must hold.

What you will learn
Workshop
A simulated schedule with p stages and m micro-batches; a forward pass takes one time unit and a backward two, and a stage can start a pass only when the previous (or, for backward, the next) stage has finished the same micro-batch. Idle fraction measured from the schedule; compare the formula (p − 1)/(m + p − 1).

The bubble and the micro-batches

With 4 stages and 8 micro-batches, the 1F1B schedule leaves the devices idle 27.3% of the time, the same as GPipe (27.3%) and exactly (p − 1)/(m + p − 1). What differs is memory: stage 1 under GPipe holds the activations of all 8 micro-batches, under 1F1B only 4. With 16 stages and 8 micro-batches the bubble swells to 65.2%; with 32 micro-batches it falls to 31.9%. Deep pipelines need many micro-batches — which means a large global batch, or few data-parallel replicas.

Two refinements matter at scale. The interleaved schedule of Megatron-LM gives each device several non-adjacent chunks of layers, cutting the bubble by that factor at the cost of more messages; Llama 3 used 16 pipeline stages with such a schedule, tuned so stages with the embedding and output layers carry fewer Transformer layers. DeepSeek-V3’s DualPipe feeds micro-batches from both ends of the pipeline and overlaps their computation with the communication of expert parallelism. Both reduce the bubble without changing the arithmetic of the model.

Pipelines also complicate everything around the model: the stage holding the output layer needs the loss, the tied embedding (if any) lives on two stages, and uneven layers make the slowest stage set the pace.

Terms

Pipeline stage
A contiguous block of layers placed on one device or group.
Micro-batch
One of the pieces a step’s batch is split into to fill the pipeline.
Bubble
Idle time at the start and end of a step while the pipeline fills and drains.
1F1B
A schedule alternating one forward and one backward pass once the pipeline is full.
Pipelines pass small activations between stages over slow links, at the price of a bubble (p − 1)/(m + p − 1) that only many micro-batches shrink.

Sources

  1. Huang et al., “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism”, NeurIPS 2019.
  2. Narayanan et al., “PipeDream: Generalized Pipeline Parallelism for DNN Training”, SOSP 2019.
  3. Narayanan et al., “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM”, SC 2021 (interleaved schedule).