Vol. XVI · Kernels and Efficiency Plate 2 of 8kernel fusion, element-wise operations, memory traffic, launch overhead, RMSNorm, SwiGLU, CUDA graphsCE

Kernel Fusion.

Five small kernels read and write the tensor five times. One fused kernel reads it once and writes it once.

RMSNorm is a handful of element-wise steps: square, average, add ε, reciprocal square root, multiply by the input and by a learned scale. Written as separate operations, each is its own kernel that reads its input from HBM and writes its output back — for a tensor of S BF16 values, 4S bytes per step. None of them does enough arithmetic to be anything but memory-bound (plate XIII.3), so the chain’s time is its total traffic. A fused kernel loads each value once, does every step in registers, and stores the result once: 4S bytes in all.

Each kernel launch also costs a few microseconds of fixed overhead. For large tensors the bytes dominate and fusion saves almost the full factor of the chain length; for small ones — decoding a single token, say — launches dominate, which is what CUDA graphs and compilers that capture whole graphs attack. The workshop prices a chain of k element-wise operations both ways.

Below: left, the time of the chain unfused and fused against the tensor size. Right, the bytes each moves at the chosen size.

What you will learn

Explain why element-wise kernels are memory-bound.
Compute the traffic of a chain unfused and fused.
Explain the role of launch overhead.
Name the operations that are commonly fused.

Workshop

A chain of k element-wise operations on S BF16 values (each kernel reads 2S bytes and writes 2S), at the H100’s 3.35 TB/s, with the chosen fixed launch overhead per kernel. Arithmetic is ignored: at a few FLOPs per value it is far below the ridge point.

Read once, write once

A five-step chain on 33 million values — one 8 192-token sequence of width 4 096 — takes 220 µs as separate kernels and 44 µs fused: 5.00 times faster, almost the full factor of five, because each unfused step re-reads and re-writes the tensor. On a tensor of 4 096 values — one token’s activations during decoding — both are dominated by launches: unfused 20.0 µs, of which 99.9% is launch overhead, against 4.0 µs fused. A single operation gains nothing (1.00×).

This is why norms, activation functions such as SwiGLU’s SiLU-and-multiply, residual additions, bias additions, rotary embeddings and dropout are almost always fused, by hand or by a compiler (plate XVI.5), and often fused into the epilogue of the preceding matrix multiply so the result never makes a separate trip. For decoding, where every kernel is tiny, capturing the whole step as a CUDA graph removes most of the launch cost by replaying a recorded sequence of launches.

Fusion changes how many times data crosses the memory hierarchy, not how much arithmetic is done — so it helps exactly the operations the roofline puts below the ridge point, and does nothing for a large matrix multiply.

Terms

Kernel fusion: Combining several operations into one kernel so intermediate results stay on chip.
Launch overhead: The fixed cost of starting a kernel, a few microseconds.
Epilogue fusion: Applying an element-wise operation to a matrix multiply’s output before it is written.
CUDA graph: A recorded sequence of kernel launches replayed with little CPU overhead.

Fusing a chain of k element-wise kernels cuts its memory traffic about k-fold; for tiny tensors launch overhead dominates, and graphs remove it.

Sources

NVIDIA, “CUDA Graphs” and CUDA C++ Programming Guide (kernel launch overhead).
PyTorch documentation, torch.compile (TorchInductor fusion).