Sandia's Liquid-Cooled Data Center Boosts Efficiency and Resiliency (FEMP success story, DOE/GO-102021-5582, July 2021)

Overview

The document describes the vacuum-operated liquid cooling system supplied by Chilldyne that Sandia chose for the 650 kW Attaway supercomputer installed in 2019. Water flows through flexible tubes and pipes in the compute racks, and at the end of the rows cooling distribution units (CDUs) with vacuum pumps control water flow and rates. Warm water is used instead of chilled water to save energy. The water-based system handles 70% of cooling; fans in each node provide the remainder, cooling low-power components and acting as a secondary cooling source.

The main benefit of vacuum is safety: a hose puncture draws air in instead of letting water leak out. Operating under vacuum also permits flexible hoses in place of the rigid pipes that positive-pressure systems require, which gives design flexibility. For resilience, 24 compute racks are served by three CDUs, each normally feeding eight racks. Six fail-over valves, each feeding four racks and fed from two different CDUs, let the system switch automatically; if one CDU fails or is taken down for service, the remaining two can cool all 24 racks.

Sandia researchers tested a sudden change from idle to full load and operation at full load without water cooling. The system responded to sudden load changes without throttling any nodes, and with water cooling removed it ran for a short time on fan air before throttling. CPUs stayed below 50 °C (122 °F), and each CDU draws 3.4 kW at full load, about 1.6% of total system power. Earlier water-cooled designs shut down completely on loss of water flow. A 1,000 kW system, Manzano, using the same vacuum cooling entered production in early 2021. This is a short case study; the detailed sequence of operation is in a separate Sandia test report.

Key points

1. The Attaway supercomputer installed in 2019 is a 650 kW HPC system; liquid cooling handles about 70% of its heat with node fans handling the rest.
2. 24 compute racks are served by three vacuum CDUs (eight racks each normally) with six fail-over valves, each feeding four racks from two different CDUs, so two CDUs can cool all 24 racks.
3. Operating under vacuum means a hose puncture draws air in rather than leaking water, and permits flexible hoses instead of rigid piping.
4. The system kept CPUs below 50 °C (122 °F); each CDU draws 3.4 kW at full load, about 1.6% of total system power.
5. With water cooling removed at full load, the system ran on fan air for a short time before throttling; it handled idle-to-full load steps without throttling.
6. Earlier water-cooled system designs shut down completely on loss of water flow; this system continues briefly on fan air.
7. A follow-on 1,000 kW HPC system (Manzano) using the same vacuum liquid cooling entered production in early 2021.
8. With this system Sandia's Albuquerque campus earned a DOE Sustainability Award and LEED v4 Gold BD+C Data Center certification in 2020; 12 months of leak-free operation are reported.

Using it in design

• Consider negative-pressure (vacuum) distribution as an option for reducing leak risk in direct liquid cooling; flexible hoses give layout flexibility.
• Arrange CDU redundancy as in this scheme: fail-over valves feeding rack groups from two different CDUs let the remaining units cool all racks on loss or maintenance of one CDU; size capacity for that scenario.
• Determine through testing how long node fans and air-side capacity can serve as secondary cooling on loss of water cooling, and how throttling behaves.
• Account for the liquid/air split (about 70% and 30% here) and CDU power (about 1.6% at full load) in the thermal budget; warm water reduces the need for chilled water.
• Add idle-to-full load steps and loss-of-cooling-water scenarios to the acceptance test specification.

Who should read it

Mechanical engineers designing direct liquid cooling and high-density HPC installations, and designers who want a real reference for CDU redundancy.

Document contents

1. Vacuum System Delivers Safe, Efficient Liquid Cooling
2. Designed for Resilience
3. Positive Performance and Striking LEED Gold
4. Overcoming Fears About Liquid Cooling
5. Innovation in HPC
6. Learn More

Read the document

Publication details

Authors

U.S. DOE Federal Energy Management Program (FEMP)

Publisher

U.S. DOE FEMP

Year

2021

Document language

English

Source type

Government / public

Access

Open licence — downloadable

Licence

U.S. public domain — U.S. Government work; free to reuse

Licence evidence

www.energy.gov

Citation