Revealing Power, Energy and Thermal Dynamics of a 200PF Pre-Exascale Supercomputer (SC21)

Overview

This paper, by Woong Shin, Vladyslav Oles, Ahmad Maroof Karimi, J. Austin Ellis and Feiyi Wang, was presented in the State of the Practice track at SC21 (ACM/IEEE Supercomputing 2021). ORNL analysed a full year (2020) of 1 Hz telemetry from all 4,626 Summit nodes together with facility meters, job logs and GPU failure logs.

The paper characterises system-level power swings, job power profiles and energy efficiency, and how the medium-temperature-water cooling plant and its PUE respond to weather and to the large, fast load changes of HPC jobs. It shows evaporative cooling carrying most of the year, with chilled-water trim in summer. Summit is a 13 MW peak system in a 20 MW facility, idling around 2.5 MW and typically operating between 5 and 6 MW.

The cooling plant supplies medium-temperature water at about 20 C (70 F) from a central plant with 8 cooling towers and 5 chillers; the chillers only trim supply temperature when the towers cannot. Average PUE is 1.11 thanks to evaporative cooling, rising to about 1.22 in summer when chilled water is used, and maintenance drove a 1.3 excursion. PUE is affected not only by weather but also by large workload power extremes.

The dataset covers more than 100 metrics per node at 1 Hz over 840k jobs and 250k GPU failure logs, an aggregated stream of about 460k metrics per second. The sum of node power differed from facility switchboard meters by about -129 kW on average, quantifying metering gaps. As a limit, the data are valid for a single supercomputer and climate and should not be generalised directly to commercial data centres.

Key points

1. Summit: 13 MW peak system in a 20 MW facility; idle around 2.5 MW, typical operation between 5 and 6 MW.
2. Medium-temperature water supplied at about 20 C (70 F) from a central plant with 8 cooling towers and 5 chillers; chillers only trim supply temperature when towers cannot.
3. Average PUE 1.11 thanks to evaporative cooling, rising to about 1.22 in summer when chilled water is used; maintenance drove a 1.3 excursion.
4. Dataset: more than 100 metrics per node at 1 Hz over 840k jobs and 250k GPU failure logs; aggregated stream about 460k metrics per second.
5. Sum of node power differed from facility switchboard meters by about -129 kW on average.
6. PUE is affected not only by weather but also by large workload power extremes.
7. Full year 2020, 4,626 nodes; SC21 State of the Practice track.

Using it in design

• Let designers enter a seasonal cooling mode split (towers vs chiller trim) in energy models rather than a single PUE.
• Consider large, fast IT load swings (several MW) when sizing upstream equipment and the grid interface.
• Account for the gap between metered and summed power (metering gap) in monitoring design.
• Assess separately the effect of maintenance periods on PUE for medium-temperature water cooling.

Who should read it

Engineers designing liquid-cooled HPC and AI facilities, energy modellers and those studying the effect of power swings on infrastructure.

Related standards

• ISO/IEC 30134-2 (PUE)

Publication details

Authors

Woong Shin; Vladyslav Oles; Ahmad Maroof Karimi; J. Austin Ellis; Feiyi Wang

Publisher

ACM/IEEE (SC Conference)

Year

2021

Document number

SC21 State of the Practice paper pap343; OSTI 1833956

Event

SC21 (ACM/IEEE Supercomputing 2021) - State of the Practice track

Document language

English

Source type

Scientific

Access

Free to read

Licence

Copyright retained; citation only — the publisher’s terms apply to the original document

Citation