LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models

Overview

The carbon footprint of LLMs, encompassing operational and embodied emissions from training, inference, experimentation and storage, is a significant concern, and accurately estimating the carbon impact of emerging LLMs before training relies heavily on GPU usage. Of existing tools only mlco2 can predict the footprint of new neural networks prior to physical training, but it cannot extend to dense or MoE LLMs, disregards critical architectural parameters, focuses solely on GPUs and cannot model embodied carbon. LLMCarbon closes these gaps.

Operational carbon comes from FLOP counts, a hardware-efficiency model driven by the parallelism setting, device power, PUE and grid carbon intensity: operational energy = Σ(P_i × eff_i × n_i × t_i) × PUE, multiplied by carbon intensity. Embodied carbon is computed as CO2eq_chip = area × CPA (carbon per area), amortised as Σ(t_i/lifetime_i × CO2eq_chip_i); total carbon = operational + embodied. Embodied factors used: CPU (TSMC 16 nm, 147 mm²) 1 kgCO2/cm²; DRAM (Micron 18 nm) 0.4 kgCO2/GB; SSD (Samsung 20 nm) 0.018 kgCO2/GB; TPUv3 (16 nm, 700 mm²) 1 kgCO2/cm²; TPUv4 (7 nm, 400 mm²) 1.6 kgCO2/cm²; V100 (12 nm, 815 mm²) 1.2 kgCO2/cm².

Embodied validation was on Meta XLM training (512 V100 GPUs in 64 8-GPU servers, each with a 32 TB SSD and 256 GB DRAM). Per-unit embodied carbon: GPU 9.78 kgCO2eq, CPU 1.47, SSD 576, DRAM 102.4; motherboard, chassis and PSU are taken as 15% of the total. Against published Google and Meta figures, error is 8.2% or less for training and -3.05% for embodied carbon. Embodied carbon makes up about 24%-35% of an LLM's whole life-cycle footprint, and 92%-95% in a Meta data centre where 97% of operational energy is renewable. Across accelerators (V100, H100, TPUv3, TPUv4) embodied footprints are similar because SSD and DRAM embodied carbon dominates; operational footprints fall by 71% (H100) and 41% (TPUv4) compared with V100.

Storage-phase power assumptions: 11.3 W/TB for cloud storage and about 1.48 W/TB for intra-data-centre data transfer; Meta data centres average 60% hardware utilisation over a 5-year hardware lifespan. Validation inputs used PUE of about 1.09-1.12 for Google and Meta facilities, with grid intensities from 127 gCO2eq/kWh (europe-north1, 91% CFE) to 394 gCO2eq/kWh (us-central1, 97% CFE). Limits: embodied factors rest on a single fabrication node and carbon-per-area assumption, and validation is limited to published figures.

Key points

1. The embodied model is CO2eq_chip = area × CPA, amortised as Σ(t_i/lifetime_i × CO2eq_chip_i); total carbon = operational + embodied; operational energy = Σ(P_i × eff_i × n_i × t_i) × PUE, multiplied by carbon intensity.
2. Embodied factors: CPU 1 kgCO2/cm²; DRAM 0.4 kgCO2/GB; SSD 0.018 kgCO2/GB; TPUv3 1 kgCO2/cm²; TPUv4 1.6 kgCO2/cm²; V100 1.2 kgCO2/cm².
3. For Meta XLM training (512 V100 GPUs in 64 servers, each with a 32 TB SSD and 256 GB DRAM), per-unit embodied carbon: GPU 9.78 kgCO2eq, CPU 1.47, SSD 576, DRAM 102.4; motherboard, chassis and PSU 15% of total.
4. Against published Google and Meta figures, error is 8.2% or less for training and -3.05% for embodied carbon.
5. Embodied carbon makes up about 24%-35% of an LLM's whole life-cycle footprint, and 92%-95% in a Meta data centre where 97% of operational energy is renewable.
6. Across accelerators embodied footprints are similar (SSD and DRAM dominate); operational footprints fall by 71% (H100) and 41% (TPUv4) compared with V100.
7. Storage-phase power assumptions: 11.3 W/TB for cloud storage and about 1.48 W/TB for intra-data-centre data transfer; Meta data centres average 60% hardware utilisation over a 5-year lifespan.

Using it in design

• In AI data-centre carbon estimation, compute operational (FLOPs, power, PUE, grid intensity) and embodied (chip area × CPA, memory and storage per GB) carbon together; in a renewable-powered facility embodied carbon rises to 92-95%.
• In hardware comparison, assess separately that embodied carbon is dominated by SSD and DRAM and that accelerator choice can cut operational footprint by 41-71% against V100.
• Make hardware lifetime and utilisation (for example 5 years, 60%) explicit inputs; embodied carbon is amortised over lifetime and execution time.

Who should read it

Designers, researchers and sustainability analysts estimating the operational and embodied carbon of AI clusters before training.

Document contents

1. 4 LLMCarbon (4.1 Overview, 4.2 Parameter Model, 4.4 FLOP Model, operational carbon model, embodied carbon model)
2. Validation (operational and embodied carbon against Google and Meta figures)
3. Case studies (accelerator comparison, renewable-powered data centre, storage phase)
4. Conclusion

Read the document

Publication details

Authors

Ahmad Faiz; Sotaro Kaneda; Ruhan Wang; Rita Osi; Prateek Sharma; Fan Chen; Lei Jiang

Publisher

International Conference on Learning Representations (ICLR 2024)

Year

2024

Document number

arXiv:2309.14393

Document language

English

Source type

Scientific

Access

Open licence — downloadable

DOI

10.48550/arXiv.2309.14393

Licence

CC BY 4.0 — share with attribution

Licence evidence

arxiv.org

Citation