Vol. XV · Running a Training Cluster Plate 7 of 9cost, energy, GPU-hours, megawatt-hours, PUE, carbon intensity, emissions, Llama 3.1, DeepSeek-V3CE

Cost and Energy.

GPU-hours times a price is dollars; times watts is energy; times the grid is carbon. Every step hides an assumption.

Every published training cost starts from GPU-hours. Meta’s model card for Llama 3.1 gives 1.46 million H100 GPU-hours for 8B, 7.0 million for 70B and 30.84 million for 405B, with estimated location-based emissions of 11 390 tonnes of CO₂-equivalent for all three. DeepSeek’s report gives 2.788 million H800 GPU-hours for DeepSeek-V3 and converts them to $5.576 million at an assumed rental price of $2 per GPU-hour — explicitly excluding the research and ablations before the final run.

Turning GPU-hours into money, energy and emissions takes three assumptions: a price per GPU-hour; a power per GPU (700 W at full load for these chips) times the data centre’s power usage effectiveness (PUE), the ratio of total facility power to IT power; and the carbon intensity of the electricity. The workshop makes all three adjustable and shows what each published number implies about them.

Below: left, the dollars for each published run at the chosen price. Right, energy and emissions for the selected run at the chosen PUE and grid intensity.

What you will learn
Workshop
GPU-hours from Meta’s Llama 3.1 model card and the DeepSeek-V3 report. Energy at 700 W per GPU (the H100’s and H800’s rated power) times PUE; servers’ CPUs, memory and network are not counted separately, so this is the GPUs’ share scaled by the facility overhead. Emissions at the chosen grid intensity. The cluster’s power is for 16 384 GPUs.

Three assumptions per number

Llama 3.1 405B’s 30.84 million GPU-hours cost about $61.7 million at $2 an hour, draw 23,747 MWh at a PUE of 1.1 and emit 9,499 tonnes of CO₂ on a 400 g/kWh grid. Meta’s own 11 390 tonnes for all three models, divided by their GPUs’ energy at 700 W, implies about 414 g/kWh with no facility overhead — a typical grid mix; with those assumptions the 405B run alone comes to 8,937 tonnes, close to the model card’s 8 930. A cluster of 16 384 such GPUs draws 12.6 MW, which is why Meta reports the grid noticing when they all pause at once.

DeepSeek-V3’s 2.788 million GPU-hours give its $5.58 million at the same price — a figure widely repeated as “the cost of the model”, though the report itself says it excludes prior research and experiments, and the price is a rental assumption, not what the cluster cost to build. Every published cost is the final run’s GPU-hours times someone’s assumptions; comparing two of them means checking that the assumptions match.

Market-based emissions, which credit renewable energy purchases, can be far lower than location-based ones for the same run; Meta’s model card reports zero market-based emissions for Llama 3.1. Which figure is quoted changes the answer completely.

Terms

GPU-hour
One GPU used for one hour: the unit of training compute bills.
PUE
Power usage effectiveness: total facility power divided by IT power.
Carbon intensity
Grams of CO₂ emitted per kilowatt-hour of electricity.
Location-based emissions
Emissions computed from the local grid’s average intensity.
Cost, energy and emissions are GPU-hours times an assumption — a price, a power and PUE, a grid — and published figures cover only the final run.

Sources

  1. Meta, Llama 3.1 model card, July 2024 (GPU-hours, 700 W TDP, location-based emissions).
  2. DeepSeek-V3 Technical Report, arXiv:2412.19437, Table 1.
  3. The Green Grid, “PUE: A Comprehensive Examination of the Metric”, 2012.