The Facility Perspective on Liquid Cooling: Experiences and Proposed Open Specification (SC18 Birds of a Feather)

Overview

This document is the slide set from a Birds-of-a-Feather session held on 13 November 2018 at SC18 (ACM/IEEE Supercomputing 2018) in Dallas. It was prepared for the Energy Efficient HPC Working Group and US DOE FEMP by Dale Sartor, Anna Maria Bailey, David Grant, Herbert Huber and Dave Martinez. The session addresses warm-water liquid cooling with national-lab facility managers.

After an overview of liquid-cooling options, economizer operation and heat reuse, it introduces a draft open specification for multi-vendor liquid-cooled racks developed with OCP and China's ODCC. The specification covers wetted materials to use and avoid, coolant operating ranges and quality limits, quick-connect sizes and flows, pressure and filtration. These values later informed OCP Advanced Cooling Solutions work.

As an example, the Maui HPC Center iDataPlex is 90% water / 10% air cooled, runs with water up to 44 C and uses 10 kW dry coolers instead of 100 kW chillers. The slides are session material, not a binding standard; the values reflect a draft specification.

Key points

1. Approved wetted materials include brass, 300/400 stainless, copper, nickel and chrome plating, PPS, PTFE, EPDM, nitrile, polysulfone, nylon 6, PVC and Viton; avoid aluminium, zinc (including brazes) and lead.
2. Fluid operating range 2-60 C (assumes ASHRAE W4, 2 C CDU approach, up to 13 C rise); shipping -40 to 75 C; 10+ year life with quarterly/monthly checks.
3. Water quality: pH 7-10.5, sulfide <1 ppm, sulfate <10 ppm, chloride <5 ppm, bacteria <100 CFU/mL, hardness <20 ppm as CaCO3, conductivity 0.2-20 uS/cm, TSS <3 ppm, turbidity <20 NTU.
4. Quick connects 6.35/9.53/12.70 mm with minimum Cv 0.33/1.1/1.9 and flow at least 0.75/1.7/3.0 GPM, up to about 60 psi.
5. Maximum particulate size 50 microns, optionally with bypass filtration.
6. Maui HPC Center iDataPlex: 90% water / 10% air cooled with water up to 44 C, 10 kW dry coolers instead of 100 kW chillers.
7. The draft specification was developed with OCP and China's ODCC.

Using it in design

• Give liquid-cooling components in the catalogue wetted-material lists, and flag aluminium or zinc on a loop with copper cold plates.
• Have coolant objects hold the quality limits above and filters a micron rating (default 50 um).
• Map the fluid operating range to the ASHRAE W-class and check chiller-less operation feasibility against climate data.

Who should read it

Mechanical designers preparing facility-side water loops and material specifications for liquid-cooled racks, and HPC facility managers.

Related standards

• ASHRAE Liquid Cooling Guidelines (W classes)
• OCP ACS Cold Plate Requirements

Publication details

Authors

Dale Sartor; Anna Maria Bailey; David Grant; Herbert Huber; Dave Martinez

Publisher

Energy Efficient HPC Working Group / U.S. DOE FEMP

Year

2018

Document number

SC18 BoF, 13 November 2018

Event

SC18 (ACM/IEEE Supercomputing 2018), Dallas - Birds-of-a-Feather session

Document language

English

Source type

Government / public

Access

Free to read

Licence

Copyright retained; citation only — the publisher’s terms apply to the original document

Citation