POLCA: Power Oversubscription in LLM Cloud Providers

Overview

This 2023 arXiv preprint from Microsoft Azure by Pratyush Patel and colleagues characterises the power use of LLM training and inference on DGX A100 servers and in production clusters, and uses it to motivate power oversubscription. Inference clusters show a diurnal pattern but stay well below provisioned power, while training clusters produce large, synchronised swings.

The numbers show the difference: LLM inference clusters use only up to 80% of provisioned power, whereas training clusters offer a smaller headroom of about 10% because of massive coordinated power peaks. In production clusters, training incurs power swings of up to 37.5% of provisioned power capacity, whereas inference changes by only up to 9%. GPU power constitutes on average 60% of server-level power, and peak GPU power can exceed the server's GPU TDP by up to 500 W.

At training iteration boundaries GPU power drops to 75% of TDP (RoBERTa), 50% (GPT-NeoX) or 20% (Flan-T5, roughly idle); the authors stress that the swings, not the peak, are the main challenge. GPU frequency locking and power capping are evaluated as control knobs, noting that power caps are reactive and let prompt-phase spikes through, and that out-of-band controls via the BMC are slow. For Flan-T5 and GPT-NeoX training, frequency capping reduced peak server power by 22% while impacting performance by 10%.

The proposed POLCA framework uses phase-aware frequency and power thresholds with a power-brake backstop. Simulation on production-like traces allows 30% more inference servers in the same power budget with minimal performance loss, maintaining SLOs. As a limitation, the results are specific to Azure's workloads and hardware, and the paper is a preprint.

Key points

1. LLM inference clusters use only up to 80% of provisioned power; training clusters offer about 10% headroom because of coordinated peaks.
2. In production clusters training incurs power swings of up to 37.5% of provisioned power capacity; inference changes by only up to 9%.
3. GPU power is on average 60% of server power, and peak GPU power can exceed the server's GPU TDP by up to 500 W.
4. At training iteration boundaries GPU power drops to 75% of TDP (RoBERTa), 50% (GPT-NeoX) or 20% (Flan-T5); the swings are the main challenge.
5. For Flan-T5 and GPT-NeoX training, frequency capping reduced peak server power by 22% while impacting performance by 10%.
6. Power caps are reactive and let prompt-phase spikes through; out-of-band controls via the BMC are slow.
7. POLCA allows 30% more servers in the same inference GPU cluster while maintaining SLOs.

Using it in design

• In the power planner, apply different diversity and oversubscription factors by workload: inference rows can be provisioned below sum-of-nameplate (for example peak about 80% of provisioned), while training rows should assume near-coincident peaks with only about 10% headroom.
• Expose a 'power oversubscription' option that requires a capping or brake mechanism and telemetry, and report the extra servers deployable.
• Account for GPU peaks above TDP when sizing PSUs and rack PDUs, and flag mixed training and inference rows as a risk.

Who should read it

Electrical designers, capacity planners and operators setting power allocation and oversubscription strategy for AI clusters.

Publication details

Authors

Pratyush Patel; Esha Choukse; Chaojie Zhang; Íñigo Goiri; Brijesh Warrier; Nithish Mahalingam; Ricardo Bianchini

Publisher

arXiv preprint (Microsoft Azure)

Year

2023

Document number

arXiv:2308.12908

Document language

English

Source type

Scientific

Access

Free to read

DOI

10.48550/arXiv.2308.12908

Licence

CC BY-NC-ND 4.0 — share unaltered with attribution for noncommercial purposes

Citation