Design Guidelines for Immersion-Cooled IT Equipment, Revision 1.0 (December 2020)

Overview

The paper addresses server designers, integrators and operators and is licensed CC BY-SA 4.0. It stresses that nearly all computing equipment is built for air, and that immersion demands separate attention to materials and fluid handling. It is organised in five parts: material compatibility, thermal design, mechanical design, electronic design and software. It names the OCP ACS Immersion Requirements document as required reading alongside it. It concentrates on immersion-ready IT hardware rather than facility sizing, and gives few hard numeric limits.

The material-compatibility chapter notes that in an air-cooled hall the risks are particulates and corrosive gases, whereas in immersion the equipment itself can be the contaminant source. Label adhesives and inks, EPDM seals in capacitors, some cable jackets, heat-shrink, battery materials, unsealed relays and thermal pastes are on the problem list, with fixes such as compatible materials, indium foil as a TIM, or helium-sealed drives instead of air-filled ones. Cable jackets and heat-shrink can degrade especially at 60 C and above.

Test methods depend on the fluid. For hydrocarbon single-phase immersion, ASTM D471 and D2240 soak tests and thermal aging are suggested. Clean fluorochemicals barely interact with polymers, but they saturate at a few parts per million with contaminants such as DOP from PVC wire insulation or silicone oils, and pass them on to other materials, which the paper calls secondary incompatibility. In two-phase systems, oils build up on boiling surfaces like limescale in a kettle. The Soxhlet extraction test is preferred because it separates absorbed from extracted mass; a mass loss below 2% indicates good compatibility, and a 48-hour test gives worst-case results.

In the thermal chapter, single-phase fluids range from about 1300 J/kgK (fluorocarbon) to 2300 J/kgK (hydrocarbon) specific heat, against 1000 for air and about 4180 for water. Because of density, energy absorbed per unit volume is nearly the same for hydrocarbons and fluorocarbons. Immersion heat sinks need a wider fin pitch because of viscosity. In two-phase immersion, boiling enhancement coatings are advised for bare-die devices of about 30 W and above (100-500 micron micro-porous copper, up to 15 times the heat-transfer coefficient of a smooth surface), and have been applied to 750 W ASICs with indium TIM.

Mechanical and electronic sections cover 19-inch and 21-inch chassis, a recommended minimum gap of 0.5 mm between servers in open baths, handles and reinforcement for vertical pulling, server mass preferably not above 34 kg and two-person handling above 18 kg. Air-based safeguards such as fan detection and thermal shutdown must be disabled, and BIOS or custom firmware should provide an immersion mode. For signal integrity, the fluid's dielectric constant lowers microstrip impedance, slightly raises insertion loss and reduces far-end crosstalk; eye-diagram and S-parameter checks are advised. An appendix lists common steps for retrofitting air-cooled hardware.

Key points

1. Cable jackets, heat-shrink and other polymers can degrade in dielectric fluids, especially at 60 C and above, and contaminate the fluid; replace them with compatible materials.
2. In Soxhlet extraction testing, a mass loss below 2% indicates good material compatibility; the 48-hour test gives worst-case results.
3. Clean fluorocarbon fluids are compatible, but they can carry hydrocarbon contaminants from PVC and silicone between materials, causing secondary incompatibility.
4. Specific heat: fluorocarbon about 1300, hydrocarbon 2300, air 1000, water about 4180 J/kgK; volumetric heat capacity is nearly equal for hydrocarbons and fluorocarbons.
5. Boiling enhancement coatings suit bare-die devices of about 30 W and above in two-phase immersion; they are 100-500 micron micro-porous copper and can raise the coefficient up to 15 times.
6. In single-phase tanks, in-chassis PSUs must be fully immersed and placed low, near the tank bottom, so they do not preheat downstream components.
7. Single-phase baths stratify into bottom (T0), middle (T1) and top (T2) zones; two-phase tanks are isothermal and need spacing for vapour escape.
8. Directly immersed storage is limited to SSD/NVMe and sealed helium drives.
9. Immersion lowers microstrip impedance, slightly increases insertion loss and reduces far-end crosstalk, so PCB impedance specifications should account for the fluid.
10. Server mass is preferably kept to 34 kg or less, with two-person handling above 18 kg; hydrocarbon fluids are slippery, so handling needs further study.

Using it in design

• Compare the bill of materials of a server to be immersed (cable jackets, heat-shrink, EPDM capacitors, CMOS battery, relays, thermal paste) against the paper's problem table and obtain a compatibility statement from the fluid supplier for each item.
• Set the test plan by fluid type: ASTM D471/D2240 soak and thermal aging for hydrocarbons, Soxhlet extraction with the 2% mass-loss threshold and filtration sizing for fluorocarbons.
• In single-phase tank layout, place high-TDP GPUs and PSUs in the lower zone and heat-tolerant components in the upper zone, and check preheating of components downstream of each PSU.
• Write per-server mass limits (preferably 34 kg), handles, guides and reinforcement for vertical pulling into the rack and tank specification, and require top access to cables.
• Require suppliers to allow fan detection, air-based thermal shutdown and temperature thresholds to be disabled or switched to an immersion mode, and add eye-diagram and S-parameter measurements in the fluid to acceptance testing.

Who should read it

Hardware engineers designing immersion-cooled servers and racks, integrators, and data centre operators selecting immersion systems.

Document contents

1. License
2. Executive Summary
3. Abstract
4. Introduction
5. Material Compatibility
6. Single-Phase Immersion Cooling (Hydrocarbons)
7. Single-Phase Immersion Cooling (Fluorochemicals)
8. Two-Phase Immersion Cooling (Fluorochemical Fluids)
9. The Soxhlet Extraction Material Compatibility Test
10. Thermal Design
11. Two-Phase Boiler Assembly
12. Component placement for Single Phase and Two Phase fluids
13. Mechanical Design (chassis, power supply, storage, optical cabling)
14. Electronic Design Guidelines (signal integrity)
15. Adjustable temperature settings, fan control, IPMI and firmware support
16. Appendix: Common practices for retro-fitting air-designed IT equipment
17. Appendix: Glossary
18. Appendix: References

Read the document

Publication details

Authors

Lesya Dymyd; Leonard Ciubotaru; Michael Helezen; Jimil M. Shah; Lentis Pai; Rolf Brink; Rick Payne; Jessica Gullbrand; Nigel Gore

Publisher

Open Compute Project Foundation

Year

2020

Document language

English

Source type

Industry body

Access

Open licence — downloadable

Licence

CC BY-SA 4.0 — share with attribution under the same licence

Licence evidence

www.opencompute.org

Citation