"Hot" for Warm Water Cooling (SC11 State of the Practice paper, LBNL-5128E)

Overview

This paper is a State of the Practice paper prepared for SC11 (ACM/IEEE Supercomputing 2011) in Seattle, published by the Energy Efficient HPC Working Group and Lawrence Berkeley National Laboratory (LBNL-5128E). Its authors are Henry Coles, Michael Ellsworth and David J. Martinez. The paper offers a clear worked method for building up a supply temperature from climate data.

The method uses 99.6% design wet- and dry-bulb conditions for 15 US national-laboratory sites. Across these sites, dry bulb is at or below 37.5 C and wet bulb at or below 26.5 C for 99.6% of the year, the remaining 0.4% being about 35 hours. Component approach temperatures are stacked for cooling-tower and dry-cooler chains: cooling tower 3.8 C, dry cooler 5.5 C, plate-and-frame heat exchanger 1.7 C and CDU 2.8 C, plus pre-heat and chip-level allowances. The result feeds either air-cooled or direct-liquid-cooled IT, and the remaining margin to a reference CPU's case limit is checked.

The results are as follows. A cooling tower at 26.5 C wet bulb gives a 32 C building supply; with direct liquid cooling the CPU case reaches about 52 C, a 26 C margin to a 77.6 C limit, whereas with air cooling the margin is 14 C. A dry cooler at 37.5 C dry bulb gives a 43 C supply; the margin is 12 C with direct liquid cooling but only 3.4 C with air at the chip. The paper recommends 32 C for cooling-tower-based supply and 43 C for dry-cooler-based supply, which fed into ASHRAE's W-class liquid cooling guidelines.

In the solutions considered, about 90% or more of the heat is typically removed by liquid. The guideline parameters identified for future work include minimum and maximum supply temperature, minimum return rise, water quality, percent of heat to liquid, maximum static pressure, minimum differential pressure and rate of change of supply temperature. As a limit, this is a 2011 study based on US sites and a specific reference CPU; the method is flexible but the numbers must be recalculated for another climate and hardware.

Key points

1. Design conditions across the 15 sites: dry bulb at or below 37.5 C and wet bulb at or below 26.5 C for 99.6% of the year (the remaining 0.4% is about 35 h).
2. Approach temperatures assumed: cooling tower 3.8 C, dry cooler 5.5 C, plate-and-frame HX 1.7 C, CDU 2.8 C, plus pre-heat and chip-level allowances.
3. Cooling tower at 26.5 C wet bulb gives 32 C building supply; direct liquid cooling puts the CPU case at about 52 C, a 26 C margin to 77.6 C; with air cooling the margin is 14 C.
4. Dry cooler at 37.5 C dry bulb gives 43 C supply; margin is 12 C with direct liquid cooling but only 3.4 C with air at the chip.
5. Recommendation: 32 C for cooling-tower-based and 43 C for dry-cooler-based supply, which fed into ASHRAE's W-class guidelines.
6. About 90% or more of heat is typically removed by liquid in the solutions considered.
7. Parameters identified for future work: min/max supply temperature, minimum return rise, water quality, percent of heat to liquid, max static pressure, min differential pressure, rate of change of supply temperature.

Using it in design

• Implement this stack-up as a design check: from the site's 99.6% design wet and dry bulb plus per-component approaches, compute the achievable chiller-less supply temperature.
• Compare the result with the IT liquid class, for example warn if a dry-cooler design is paired with air-at-chip IT needing below 43 C.
• Report the CPU case temperature margin separately for direct liquid and air cooling.
• Recalculate the figures for your own climate and hardware.

Who should read it

Mechanical designers and HPC facility engineers sizing the heat rejection chain for liquid-cooled IT from climate data.

Related standards

• ASHRAE Liquid Cooling Guidelines for Datacom Equipment Centers (W classes)

Publication details

Authors

Henry Coles; Michael Ellsworth; David J. Martinez

Publisher

Energy Efficient HPC Working Group / Lawrence Berkeley National Laboratory

Year

2011

Document number

LBNL-5128E

Event

SC11 (ACM/IEEE Supercomputing 2011), Seattle

Document language

English

Source type

Government / public

Access

Free to read

Licence

Copyright retained; citation only — the publisher’s terms apply to the original document

Citation