Data Center Liquid Distribution Guidance & Reference Designs, Revision 1.0 (January 8, 2024)

Overview

The paper observes that liquid cooled IT has long been used in HPC, usually installed at initial construction, but adding or updating it in later phases is becoming a lifecycle requirement. Because IT refresh happens every 3-5 years, preserving pipe paths and connection points is presented as the way to defer capital cost and avoid disruption. Planning areas are piping sizing and routing, BIM-based reference designs, temperature control, isolation and thermal ride-through, service level agreements, and balancing cost and risk through design, commissioning and procedures. Options for existing facilities include tying into the chilled water system, or bypassing the chiller plant with dry coolers or a closed condenser water loop.

Case 1 covers connection to an existing facility water system, for example by swapping a CRAH or adding tap-offs. Pipe sizing follows Q = volumetric flow x delta T x density x specific heat, and water carries around 3,300 times more heat per volume than air. Glycol and dielectric fluids lower heat capacity; the pipe size versus kW table is conceptual only. Pipe movement matters: the swing from install to peak can exceed 30 °C, and HDPE moves about 7 times as much as stainless steel. CDU needs differ by technology. Cold plate and immersion usually want a CDU to control pressure, quality and temperature, door heat exchangers can often connect directly, and a recirculating pump with a control valve can substitute for a CDU without loop isolation.

Case 2 covers elevated temperature loops above 30 °C. The paper cites ASHRAE TC 9.9's 2021 guidelines with six liquid cooling classes, says cold plate and immersion loops at 35-60 °C are common and can be served by dry heat emission without evaporation, and notes that condensation is not a concern because the highest ASHRAE dew point is 28 °C for Class C and ASHRAE recommends a maximum of 15 °C. Liquid above 50 °C (122 °F) makes heat reuse practical, for example for greenhouses, pools or district heating, improving ERE. A CDU at the facility boundary may still help separate the loop from external liquids, temperatures and pressures.

The risk sections cover thermal ride-through and loss-of-flow, where pump redundancy, backup power and extra loop volume near the IT are recommended. They propose an SLA boundary at the CDU, since ownership of TCS water quality between operator, vendor and tenant is a common problem. FMEA is described through severity, probability and detection of failure, with measures such as locating piping away from or below IT, verifying connection installation and leak detection on threaded, flanged, crimped or fused joints. Tier III and IV implications, SOP, MOP and EOP content, commissioning, BIM content and prefabrication are covered, with appendices on distribution loops, keys to success, FMEA and closed-loop cleaning. The document states it is not comprehensive and has few hard limits.

Key points

1. IT refresh occurs every 3-5 years, so preserving pipe routes and tap-off points lets liquid cooling be added later with minimal cost and disruption.
2. Heat transfer is Q = volumetric flow x delta T x density x specific heat; by volume water carries around 3,300 times more heat than air.
3. Glycol additives and dielectric immersion fluids reduce heat capacity, so they need more volume to move the same heat.
4. Temperature swing from installation to peak operation can exceed 30 °C; HDPE has about 7 times the thermal movement of stainless steel.
5. Cold plate and immersion loops at 35-60 °C are common and can be served by dry heat rejection without evaporation.
6. Liquid above 50 °C (122 °F) is far better for heat reuse than 35 °C (95 °F) air, and improves ERE.
7. ASHRAE TC 9.9 (2021) lists a maximum dew point of 28 °C for Class C and recommends a maximum of 15 °C; above 15 °C liquid poses minimal condensation risk.
8. Door HX systems can often connect directly to facility water; cold plate and immersion generally use a CDU for pressure, quality and temperature separation.
9. The CDU is a logical SLA boundary, with thermal metering of flow, temperature and pressure on the TCS side.
10. Tier III concerns cooling supply to the tank, door HX or cold plate, not liquid distribution inside the IT; MTTR is a key availability factor.

Using it in design

• Reserve space for tap-off valves, pipe paths with straight runs, valve and fitting replacement, drainage and condensation collection in the original design.
• Use the Q equation to estimate pipe size against kW and delta T, and confirm it with a licensed engineer, since the paper's table is for concept comparison only.
• Decide per IT technology whether a CDU, recirculating pump or direct connection is used, and place CDUs where maintenance, weight and wet weight can be accommodated.
• Plan thermal ride-through and a loss-of-flow response: redundant pumps, backup power, and extra loop volume nearest the IT.
• Define the SLA boundary and metering at the CDU, and state who maintains and warrants TCS water quality.
• Run an FMEA on connections and require inspection or leak detection according to the joint type and the severity of the location.

Who should read it

Facility designers, operators and colocation providers planning for future liquid cooled IT, and IT users who need to understand facility-side liquid distribution.

Document contents

1. Executive Summary
2. 1. Introduction
3. 2. Compliance with Open Compute Project Tenets
4. 3. Design Considerations
5. 3.1 Case 1 - Connection to Existing FWS System
6. Planning for Pipe Connections
7. Pipe Route Planning
8. Pipe Movement
9. Pipe Material Selection
10. Coolant Distribution Unit (CDU) Planning
11. 3.2 Case 2 - Elevated Temperature Loop (>30C) Considerations
12. 3.3 Service Level Agreements (SLA) and Risk Considerations
13. 3.4 Risk & Reliability Considerations
14. 3.5 VDC Enablement of Liquid Cooled ITE in Data Center Life-Cycle
15. 4. Conclusion
16. Appendix B. Cooling Distribution Loops
17. Appendix D. Probability Risk Factor Reduction (FMEA)
18. Appendix E. Closed Loop Cleaning Best Practices

Read the document

Publication details

Authors

Don Mitchell; John Menoche; John Gross; Vali Sorell; John Musilli

Publisher

Open Compute Project Foundation

Year

2024

Document language

English

Source type

Industry body

Access

Open licence — downloadable

Licence

CC BY-SA 4.0 — share with attribution under the same licence

Licence evidence

www.opencompute.org

Citation