A Power-Measurement Methodology for Large-Scale, High-Performance Computing (ICPE 2014)

Overview

This paper describes the three-level power-measurement methodology developed through the EE HPC WG with Green500, Top500 and Green Grid for benchmarking supercomputer energy efficiency. Each level tightens the fraction of the machine measured, the time span and granularity, the subsystems included and where in the power chain the meter sits. It is the reference for how much of a system must be metered to make an efficiency claim.

Level 1 uses one-second samples averaged over at least 20% of the core run, with at least 1/64 of the compute nodes or 1 kW measured and only compute nodes required. Level 2 spreads measurements across the full run, measures at least 1/8 of compute nodes, and measures or estimates other participating subsystems. Level 3 continuously integrates energy over the whole system, takes at least 10 readings in the core phase, reports idle power and meters upstream of power conversion or measures the conversion loss.

Case studies from LRZ, Argonne and Calcul Quebec show the practical difficulties and how much results differ between levels. For example, SuperMUC PDUs sample at 120 Hz with 60 s averages, and its Level 1 result was 1055 MFlops/W versus 905 MFlops/W at Level 3. The authors recommend measuring at some level rather than waiting for Level 3 instrumentation. The paper focuses on HPC; the method does not replace general data centre PUE measurement.

Key points

1. Level 1: one-second samples averaged over at least 20% of the core run; at least 1/64 of compute nodes or 1 kW measured; only compute nodes required.
2. Level 2: measurements spread across the full run; at least 1/8 of compute nodes; other participating subsystems measured or estimated.
3. Level 3: continuously integrated energy over the whole system; at least 10 readings in the core phase; idle power reported; metered upstream of power conversion or conversion loss measured.
4. SuperMUC PDUs sample at 120 Hz with 60 s averages; Level 1 result 1055 MFlops/W versus 905 MFlops/W at Level 3.
5. Authors recommend measuring at some level rather than waiting for Level 3 instrumentation.
6. Case studies come from LRZ, Argonne and Calcul Quebec.
7. Related references: EE HPC WG Power Measurement Methodology, ISO/IEC 30134-2.

Using it in design

• In the metering design module, offer Level 1/2/3 templates that place meters (for example upstream of PSUs/PDUs) and set the required coverage and sampling.
• Warn when efficiency metrics are claimed from partial-sample (Level 1) metering.
• State clearly in reports which measurement level was used.

Who should read it

Engineers who measure and report energy efficiency of HPC and AI clusters.

Related standards

• EE HPC WG Power Measurement Methodology
• ISO/IEC 30134-2

Publication details

Authors

Thomas R. W. Scogland; Craig P. Steffen; Torsten Wilde; Florent Parent; Susan Coghlan; Natalie Bates; Wu-chun Feng; Erich Strohmaier

Publisher

ACM/SPEC

Year

2014

Document number

ICPE 2014

Event

ICPE 2014 (ACM/SPEC International Conference on Performance Engineering)

Document language

English

Source type

Scientific

Access

Free to read

Licence

Copyright retained; citation only — the publisher’s terms apply to the original document

Citation