Vol. XXIII · Serving Plate 5 of 9latency, throughput, time per output token, batch size, service level objective, tokens per secondCE

Latency or Throughput.

More users per step: more tokens in all, fewer for each.

Serving is measured by three numbers. Time to first token (TTFT) is how long a user waits for the answer to start — prefill and queueing. Time per output token (TPOT) is the gap between tokens after that — one decode step. Throughput is the total output tokens per second across all users, which with the GPU’s price sets the cost (plate XXIII.8).

The decode batch size trades the last two against each other. A step costs at least one read of the weights; adding sequences adds their arithmetic and the reading of their caches. While the weights dominate, extra sequences are nearly free and throughput rises almost in proportion; as the caches and arithmetic grow, each step slows and each user sees fewer tokens per second. The workshop computes one decode step of Llama 3 8B on an H100 for the chosen batch size and context length.

Below: total output tokens per second against time per output token as the batch grows from 1 to 512, for the chosen context length; the point marks the chosen batch.

What you will learn

Define TTFT, TPOT and throughput.
Trace the batch-size trade-off.
See long contexts shift it.
Choose a batch under a latency target.

Workshop

One decode step of Llama 3 8B on one H100: B sequences, each holding c cached tokens. Time per output token is the step time; throughput is B divided by it; per-user speed is its inverse. Times are roofline estimates from the H100’s published specifications (989 TFLOP/s dense BF16, 3.35 TB/s), assuming 60% of peak compute and 80% of peak bandwidth are reached plus 0.2 ms per pass — not measurements.

The dial between fast and cheap

With 2 000 tokens of context, one user alone gets 159 tokens a second at 6.3 ms each. A batch of 64 delivers 5,139 tokens a second in all at 12.5 ms each — far more total for a modest slowdown per user — and a batch of 256 delivers 8,196 at 31.2 ms, about 52 times the single-user throughput. Each further doubling buys less, because the caches being read grow with the batch while the weights stay one read.

Context length moves the whole curve. At 16 000 tokens per sequence the batch of 64 delivers only 1,137 tokens a second at 56.3 ms each: reading the caches now outweighs reading the weights. Operators choose a point by a service-level objective — say, every user at least 20 tokens a second, faster than people read — and the largest batch meeting it is the cheapest way to serve. Long-context and agent workloads, with large caches per user, are expensive to serve for exactly this reason.

Users perceive the two latencies differently: a long wait for the first token feels like a slow service; a slow stream of tokens after it feels like a slow model.

Terms

TTFT: Time to first token: queueing plus prefill.
TPOT: Time per output token: one decode step.
Throughput: Output tokens per second over all users.
Service-level objective: A latency target the service promises to meet.

Batch size is the dial between latency and throughput; long contexts make every point on it slower.

Sources

Pope et al., MLSys 2023.
Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve”, OSDI 2024.