Energy Performance Evaluation of Aquila's Aquarius Fixed Cold Plate Cooling System at NREL's High Performance Computing Center (NREL/TP-2C00-73356, February 2019)

Overview

In the first half of 2018 Aquila installed its Aquarius rack, a fixed cold plate liquid-cooled design, at NREL's Energy Systems Integration Facility under a partnership with Sandia. The Sandia cluster, called Yacumama, has 36 compute nodes on Intel S2600KP boards with dual Xeon processors, 128 GB RAM, a 128 GB SSD and an Omni-Path adapter, rated at more than 40 teraflops of LINPACK while drawing under 15 kW. NREL's data center was chosen because it is built for liquid cooling and instrumented to measure flow and temperature difference. The facility targets a PUE of 1.06 or better, has no compressor-based cooling, and supplies up to 75 °F (24 °C) water with return at 95-105 °F (35-40 °C).

The rack hardware is described in detail. Each cold plate is a 2-mm laser-welded sandwich of two stainless steel plates, 21 by 28 in., holding about a pint of water. Heat sources over 2 W reach the plate through aluminum spreaders and a compliant thermal interface material. Flow is set at 1 gpm per plate and each plate can remove over 2 kW at an 85 °F input; a 12-plate manifold delivers a maximum of 12 gpm, and a 42U OCP rack can hold up to 24 plates. Power uses the OCP 12 VDC bus from a 415 VAC bulk supply of nine rectifiers giving up to 21.6 kW, which removes one conversion step and saves about 5% of load. The rack CDU draws about 750-800 W.

Testing was limited to facility-grade instrumentation and a facility supply that could not be altered, at typically about 72 °F. A discovery round raised the CDU IT supply set point from 80 to 95 °F under 100% LINPACK. Compute node power stayed about 12.6-12.7 kW, mean facility flow fell from 10.6 to 3.4 gpm, mean CPU temperature rose from 145.4 to 162.7 °F and maximum CPU temperature from 165 to 183 °F. A 48-hour test was run, but a data logger restart left a clean 20-hour window at an 85 °F set point, in which 251.1 kWh was recovered directly to water. Turbo stayed enabled and no clock throttling was seen.

Heat capture was 98.3% for the compute nodes alone, 93.4% for compute nodes plus the Powershelf (which is air-cooled, with 95% efficiency leaving 0.6 kW to air) and 86.1% when the CDU, switches and head node are included. The first two figures are upper limits because CDU heat transfer to facility water could not be measured. The cluster ran about 10 months with zero maintenance and no leaks. Installation took half a day. The authors suggest warmer supply water is possible, which would favor waste heat reuse and dry coolers, and propose moving the Btu meter closer to the CDU, adding secondary-loop thermocouples and checking flow meter calibration. It is a short early test of one vendor system.

Key points

1. Heat captured directly to water reached up to 98.3% for compute nodes only, 93.4% with the Powershelf and 86.1% with CDU, switches and head node included.
2. The 36-node cluster is rated at more than 40 teraflops LINPACK at under 15 kW.
3. Each 21 by 28 in. cold plate can remove more than 2 kW at 1 gpm and an 85 °F input; a 12-plate manifold delivers a maximum of 12 gpm; a 42U OCP rack can hold up to 24 plates.
4. Server auxiliary fans are eliminated; components over 2 W are cooled through heat spreaders and a compliant thermal interface material.
5. Bulk 415 VAC to 12 VDC conversion with an OCP bus removes one conversion step, saving about 5% of total load; nine rectifiers deliver up to 21.6 kW.
6. The rack CDU draws about 750-800 W.
7. Discovery tests raised the CDU supply set point from 80 to 95 °F; facility flow fell from 10.6 to 3.4 gpm and mean CPU temperature rose from 145.4 to 162.7 °F with no throttling.
8. The 20-hour dataset at an 85 °F set point recovered 251.1 kWh directly to water; mean facility return was 87.1 °F.
9. NREL's facility targets PUE of 1.06 or better with no compressor cooling, 75 °F supply and 95-105 °F return.
10. No maintenance and no leaks over nearly 10 months; installation took half a day.

Using it in design

• Use the measured capture fractions (98.3%, 93.4%, 86.1%) to estimate the residual air load for a fixed cold plate rack, remembering that power shelves, CDUs and switches stayed air-cooled.
• Size secondary loop flow from the per-plate figures (1 gpm, over 2 kW per plate at 85 °F input) and check manifold capacity against the plate count.
• Include the rack CDU pump power (about 750-800 W for a 36-node rack) when comparing liquid cooling against fans.
• Consider bulk AC to 12 VDC distribution as a way to remove one conversion stage, as this rack saved about 5%.
• Plan the facility loop for supply near 72-85 °F and return near 87-105 °F, which suit dry or thermosyphon coolers and heat reuse.
• Specify metering near the CDU with calibrated flow and secondary-side temperature sensors if heat capture is to be verified.

Who should read it

Designers and operators evaluating direct liquid cooling and warm-water loops for HPC, and engineers who need measured heat-capture data for a cold plate rack.

Document contents

1. Executive Summary
2. 1 Introduction
3. 1.1 Brief Aquarius Development History
4. 1.2 Aquila's Aquarius Cooling System
5. 1.3 Rack Hardware
6. 1.3.1 Electrical Distribution
7. 1.3.2 Hydronic System
8. 1.4 NREL HPC Facility Cooling Overview
9. 1.4.1 Sandia HPC Facility Cooling Overview
10. 1.5 Installation at NREL
11. 2 Test Methodology
12. 2.1 Data Collection
13. 3 Results
14. 3.1 Discovery Round of Testing
15. 3.2 Long-Duration, 20-Hour Data Set
16. 4 Conclusion
17. 5 Suggestions for Further Work

Publication details

Authors

David Sickinger; David Martinez; Bob Bolz

Publisher

NREL

Year

2019

Document language

English

Source type

Government / public

Access

Citation only

Licence

Licence conditions not established — confirm the conditions with the publisher

Licence evidence

www.nrel.gov

Citation