Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale

Overview

As research and deployment of AI grows, the computational burden to support it grows too; to train or fine-tune state-of-the-art language models some form of AI hardware acceleration is virtually a requirement. The paper studies the aggregate effect of power-capping GPUs on GPU temperature and power draw. A 60% power cap was deployed on GPUs at an academic supercomputing center. The dataset covers 123,204 GPU jobs, of which 19,676 were subject to power caps; the cluster consists of 224 nodes with two NVIDIA V100 GPUs each.

The authors apply hypothesis tests and average-treatment-effect estimates to compare capped and uncapped jobs. The estimated average treatment effect of power capping was a reduction of about 10 W in per-job GPU power draw and 4.78 °C in mean GPU temperature (95% CI -4.93 to -4.62 °C). Power-capped jobs show a smaller and more stable range of temperature fluctuations than uncapped jobs. Prior work cited: capping a model at 150 W (60% of peak) required 8.5% additional compute time but saved 12.5% of energy; another cluster capped GPUs from 400 W to 225 W while preserving 90% of performance.

The authors argue that power caps can extend hardware life and amortise embodied carbon, but warn that users may submit extra jobs to compensate for slowdowns and so erode the energy savings.

Limits: observational job-level telemetry; because capped and uncapped jobs were not randomly assigned, causal inference is limited even with matching and bias mitigation. A single centre and GPU generation (V100); results do not transfer directly to newer GPUs.

Key points

1. A 60% power cap was deployed on GPUs at an academic supercomputing center.
2. The dataset covers 123,204 GPU jobs, of which 19,676 were subject to power caps; the cluster consists of 224 nodes with two NVIDIA V100 GPUs each.
3. Prior work cited: capping a model at 150 W (60% of peak) required 8.5% additional compute time but saved 12.5% of energy; another cluster capped GPUs from 400 W to 225 W while preserving 90% of performance.
4. The estimated average treatment effect of power capping was a reduction of about 10 W in per-job GPU power draw and 4.78 °C in mean GPU temperature (95% CI -4.93 to -4.62 °C).
5. Power-capped jobs show a smaller and more stable range of temperature fluctuations than uncapped jobs.

Using it in design

• Consider a GPU power cap (for example 60% of peak) as a control in capacity and cooling planning: about 10 W per job and a 4.78 °C lower mean temperature were reported.
• Verify the performance cost of a cap (cited: 8.5% extra time for 12.5% energy saving) per workload, and allow for savings being eroded if users submit extra jobs to compensate.
• Add the potential of more stable temperatures to extend hardware life and amortise embodied carbon to the life-cycle cost calculation.

Who should read it

Operators and researchers setting GPU cluster energy and temperature management and sustainable HPC policies.

Document contents

1. I. Introduction
2. II. Related Work
3. III. Data & Methodology (A. An Empirical Analysis of Power Capping, B. Hypothesis Testing & Inference, C. Average Treatment Effect (ATE) Estimation, D. Bias Mitigation & Matching for ATE, E. Performance Impact)
4. IV. Discussion and Vision
5. V. Conclusion

Read the document

Publication details

Authors

Dan Zhao; Siddharth Samsi; Joseph McDonald; Baolin Li; David Bestor; Michael Jones; Devesh Tiwari; Vijay Gadepally

Publisher

Proceedings of the 2023 ACM Symposium on Cloud Computing (ACM)

Year

2023

Document number

arXiv:2402.18593

Pages

588-596

Document language

English

Source type

Scientific

Access

Open licence — downloadable

DOI

10.1145/3620678.3624793

Licence

CC BY 4.0 — share with attribution

Licence evidence

arxiv.org

Citation