Cold Plate Cooling Loop Requirements, Rev 2

Overview

The document is edited by Cam Turner of CoolIT Systems with contributors from Boyd, Calyos, Colder Products, Intel, Meta, Parker Hannifin, Staubli, Vertiv, ZutaCore and Veolia. It applies to rack manifold distributed liquid cooling through a technology cooling system (TCS) loop from the CDU to the rack, through the manifold and IT equipment and back, with heat passed through plate and frame heat exchangers to a facility water system that is out of scope. A qualification process is defined: a filled-out checklist goes to project leads, the submitter gives a 20-minute presentation, a 60-minute committee review follows with one week to review and one week to vote, each company has one vote and quorum is 5 votes. It covers single-phase and pool boiling cold plates only.

Technology definitions introduce cold plates, cold plate loops, hybrid and full liquid cooling, single-phase and two-phase fluids, and separate single-phase CDUs from two-phase CDUs. A two-phase CDU condenses vapor returning from the IT, buffers it in a tank and controls boiling temperature by regulating pressure of the liquid leaving, with subcooling set by facility flow. The classifications are hybrid basic, with CPU/GPU cold plates capturing roughly 65-75% of IT heat, hybrid intermediate, which adds DIMM cold plates for roughly 75-85%, and full liquid, close to 100% of components on cold plates. Liquid-cooled ready racks come with or without a door heat exchanger, and with one the room is heat neutral apart from unintentional losses.

The coolant section compares water-based, glycol-based, dielectric and two-phase refrigerant fluids in a table of pros, cons and environmental concerns. Glycol at 25% or above suppresses bacterial growth. Dielectric fluids are electrically insulating but denser and costlier, with a higher global warming potential, and the document flags PFAS concerns for some two-phase fluids while the European Chemicals Agency considers regulation. Separate chapters list parameters of importance for single-phase and pool-boiling cold plates, in-chassis hoses and tubing, quick disconnects (including maximum operating and burst pressure, in kPa), cold plate loops, rack manifolds and CDUs. Manifolds fed from the rack foot need an automatic air bleed valve at the top, which pool boiling systems do not need.

Hardware management covers sensors and Redfish telemetry, with extra sensors such as vapor pressure recommended for two-phase systems. The regulations chapter repeats a typical TCS operating pressure of 140-450 kPa (20-65 psi) for in-rack and row CDUs and the rule that loops be tested to the highest of IEC 62368-1 (3 times normal operating pressure, 2 times under abnormal and single-fault conditions) and ASME B31.3 (1.5 times design pressure). Risk management lists spill materials, containment, safety data sheets and a leak action plan, and adds a flammability check for coolants and refrigerants. A comparison metrics table captures rack width, depth, power per rack and cooling type. The document gives few hard numeric limits and defers fluid details to other OCP papers.

Key points

1. Rev 2 covers single-phase cold plates and two-phase pool boiling; flow boiling, loop heat pipes and immersion are out of scope.
2. Qualification: checklist submission, a 20-minute presentation, a 60-minute committee review, one week for questions and one for voting; each company has one vote and quorum is 5 votes.
3. Hybrid basic IT equipment (CPU/GPU cold plates) captures roughly 65-75% of IT heat, hybrid intermediate (adding DIMMs) roughly 75-85%, full liquid close to 100% of components.
4. A liquid cooled ready rack with a door heat exchanger can be room neutral; without it the heat not captured by liquid goes to room air.
5. Typical TCS operating pressure is 140-450 kPa (20-65 psi) with in-rack and row-level CDUs; facility CDUs may be higher.
6. Loops must be tested to the highest of IEC 62368-1 (3x operating, 2x abnormal and single fault) and ASME B31.3 (1.5x design pressure).
7. Glycol at 25% or above prevents bacterial growth; dielectric fluids are insulating but cost more and have higher GWP.
8. Some two-phase fluids raise PFAS concerns, and ECHA is examining regulation of refrigerants.
9. A two-phase CDU condenses vapor, buffers it and controls boiling temperature through the pressure of the liquid leaving the unit.
10. Rack manifolds fed from the rack foot need an automatic air bleed valve at the top; pool boiling systems may need occasional air purging instead.

Using it in design

• Require each vendor to fill in the parameters of importance tables for cold plates, hoses, quick disconnects, loops, manifolds and CDUs so offers can be compared.
• Classify the IT equipment and rack, and use the 65-75%, 75-85% and near 100% capture ranges to size the residual room air-cooling load.
• Set TCS design and test pressure from the highest of IEC 62368-1 and ASME B31.3, within the typical 140-450 kPa operating range.
• Choose the coolant by weighing heat transfer, additive effects, glycol concentration, dielectric cost, GWP and PFAS exposure, and specify water quality testing.
• Provide an automatic air bleed valve at the high point of foot-fed manifolds and plan leak response, spill materials and safety data.
• For two-phase systems, add sensors such as vapor pressure and check the CDU's pressure and subcooling control.

Who should read it

Cold plate, CDU and manifold vendors, rack integrators and facility designers writing or reviewing liquid cooling loop requirements.

Document contents

1. License
2. Introduction
3. Background
4. Requirements and Qualification Process
5. Technology Definitions
6. Liquid Cooling Classifications
7. IT Equipment Classifications
8. Liquid Cooled Rack Classifications
9. Cooling Liquid Selection
10. Cold Plates
11. In-chassis Hose/Tubing
12. Quick-Disconnects
13. Cold Plate Loop
14. Rack Manifolds
15. Coolant Distribution Units
16. Hardware Management
17. Sensor Requirements
18. DMTF Redfish
19. Regulations and Certifications
20. Pressure Safety Requirements
21. Risk Management
22. Comparison Metrics
23. Conclusion

Read the document

Publication details

Authors

Cam Turner; Christopher Chapman; Benjamin Sutton; Elizabeth Langer; Jessica Gullbrand; Jordan Johnson; Sean Sivipalan; Jeff Grau; Joshua Harmon; Drew Tuholski; Mick Jones; Lilach Butchmits; Shahar Belkin; Philip Yu

Publisher

Open Compute Project Foundation

Year

2023

Document language

English

Source type

Industry body

Access

Open licence — downloadable

Licence

CC BY-SA 4.0 — share with attribution under the same licence

Licence evidence

www.opencompute.org

Citation