Vol. XXIII · Serving Plate 8 of 9cost per token, GPU-hour, pricing, input tokens, output tokens, throughput, energyCE

The Cost of a Token.

Divide the price of a GPU-hour by the tokens it serves.

The economics of serving reduce to one division: what an hour of the hardware costs, over how many tokens it produces in that hour. Hardware is rented or depreciated by the GPU-hour; DeepSeek, for instance, priced its training at an assumed rental rate per H800 GPU-hour. Throughput comes from the previous plates — and it differs enormously between the two phases.

Input tokens are prefilled hundreds or thousands at a time, at the arithmetic limit; output tokens are decoded one per sequence per step, near the memory limit. API prices reflect it: Anthropic, like other providers, charges several times as much per output token as per input token. The workshop computes the hardware cost per million tokens of Llama 3 8B on an H100 at the chosen price, batch size and context length, with energy at the H100’s 700 W rating.

Below: hardware cost per million output tokens against batch size for three context lengths, with the chosen setting marked; readouts give input and output cost, their ratio and energy per thousand output tokens.

What you will learn

Compute cost per token from throughput.
Explain why output costs more than input.
See batch size and context move cost.
Estimate energy per token.

Workshop

Llama 3 8B in BF16 on one H100 at the chosen price per GPU-hour (an assumption; DeepSeek assumed $2). Output: decode at the chosen batch and context. Input: prefill of 2 048-token prompts. Energy: 700 W for the step time, divided among the batch; hardware only, excluding cooling, networking, idle time and margin. Times are roofline estimates from the H100’s published specifications (989 TFLOP/s dense BF16, 3.35 TB/s), assuming 60% of peak compute and 80% of peak bandwidth are reached plus 0.2 ms per pass — not measurements.

Output is the expensive half

At $2 an hour, decoding Llama 3 8B for one user costs about $3.49 per million output tokens: the GPU reads its weights about 159 times a second for one token each. Batching 64 users at 2 000 tokens of context brings it to $0.108; at 16 000 tokens of context the same batch costs $0.488, because reading the caches now dominates. Prefilling input costs $0.0156 per million tokens — 7 times less than batched output — because a prompt’s tokens share each read of the weights.

The pattern carries into published prices, which also cover larger models, idle capacity, redundancy, research and margin: Claude Sonnet 4 lists $3 per million input tokens and $15 per million output tokens, 5 times as much. The hardware numbers here are floors under perfect utilisation; real services run below full batch much of the day. The levers on cost are the subjects of this volume and the next: larger batches, shorter or shared contexts, cached prefixes, and smaller, quantised or speculatively decoded models (Vol. XXIV).

Reasoning models change the arithmetic of a request: thousands of hidden reasoning tokens are output tokens, decoded and billed like the visible answer.

Terms

GPU-hour: One GPU for one hour: the unit of rented compute.
Cost per token: The price of hardware time divided by the tokens produced in it.
Input and output pricing: Separate prices per million prompt and generated tokens.
Utilisation: The share of time the hardware does useful work.

Cost per token is price per GPU-hour over tokens per hour; memory-bound output costs several times compute-bound input.

Sources

DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024, Table 1 (assumed rental price).
Anthropic, “Pricing” documentation (accessed October 2026).
NVIDIA H100 datasheet (700 W).