Liquid Cooling Integration and Logistics White Paper, Revision 1.0

Overview

The document concentrates on integration and logistics of the technology cooling system (TCS) infrastructure. It defines server manufacturing levels L1 to L12: L10 is a complete server able to power on and boot, L11 is a rack with servers installed and manifolds attached, and L12 is a rack ready for deployment with a pre-determined software environment. Liquid-cooled racks include extra struts and fasteners to carry the weight of the liquid. In direct-to-chip cooling more than 70% of the heat load goes to liquid, so server fans can be reduced.

In the manufacturing flow, leak detection is done with gas (nitrogen or forming gas) at both L10 and L11, after which the system is primed with fluid of the end-use composition and put through a factory witness test. Forming gas (for example 95% nitrogen, 5% hydrogen) allows leak sources to be found with a hydrogen detector. Shipping can be filled (labelled, with fluid composition chosen against freezing) or without fluid (drained by gas and charged with nitrogen). For shipments without fluid a separate fluid recharge and trial-run room at the destination is recommended, and a check that gas has not leaked is required before filling because lost gas signals a possible leak.

Delivery, inspection (mechanical, hydraulic, electrical, network) and preparation on site are described: the rack is received against an inspection protocol, acclimatised, electrically safety-tested, gas pressure-tested with pressure decay recorded, flushed and filled at a pressure near operating pressure, with continuous venting of air. Before installation the hydraulic installation's pressure-difference inspection must be documented and heat-load dummies can be used to test-run the circuit and controls; leak detection must be planned so that leaking fluid is guided to the sensors.

Commissioning has five stages: electrical inspection, mechanical inspection, hydraulic and power circuits, controller set-up and complete start-up. Flow and pressure are raised progressively, a fluid sample is taken and alarms are tested with prescribed faults. Operation is split into monitoring, unscheduled and scheduled maintenance, and decommissioning reverses the connection order. The paper is a process guide; numeric acceptance thresholds (such as permitted pressure decay) are left to the customer and manufacturer.

Key points

1. Manufacturing levels: L10 complete tested server (boots), L11 rack with servers and manifolds attached, L12 rack ready for deployment with a software environment.
2. At L11 rack integration, servers, RDHx, RPU and manifolds are gas leak-tested before the rack is primed with end-use coolant composition for the factory witness test.
3. Racks may ship filled or drained and gas-charged; for gas-charged shipments a check that gas has not escaped is required before filling since loss of gas indicates a possible leak.
4. Coolant shipped inside L10/L11 systems should be chosen to avoid freezing during transport.
5. Manifolds and cold plates are flushed, drained and sealed with nitrogen before shipping; inert gas preserves the assembly against corrosion, oxidation and biological growth.
6. On site, gas pressure testing after transport is recommended with pressure decay recorded, followed by flushing, filling via a flushed filling pump near operating pressure and continuous venting of air.
7. Before rack installation the hydraulic installation needs a documented pressure-difference inspection; heat-load dummies can test-run the circuit and controls; leak detection must be planned so leaking fluid is guided to sensors.
8. Recommended connection sequence: hydraulic manifolds first (then vent), then network, then electrical supply; reverse order at decommissioning: power, network, then coolant.
9. Commissioning has five stages: electrical inspection, mechanical inspection, hydraulic and power circuits, controller set-up and complete start-up; start at lower flow, then raise to expected maximum conditions.
10. In direct-to-chip servers more than 70% of the heat load goes to liquid, so fans are reduced.

Using it in design

• Add mechanical, hydraulic, electrical and network inspection protocols and a recorded pressure-decay test to the delivery and commissioning plan for rack acceptance.
• For gas-shipped racks plan the pre-fill gas-loss check and a separate fill and trial-run room in the facility layout.
• In site preparation require the hydraulic pressure-difference inspection, a circuit trial with heat-load dummies and leak detection designed to guide fluid to the sensors.
• Fix the order of connection and removal: hydraulic, network, electrical on installation; electrical, network, coolant on removal.
• Put the five commissioning stages, progressive flow and pressure increase, fluid sampling and alarm tests with prescribed faults on the commissioning checklist.
• Plan a separate service area so coolant monitoring and exchange can happen outside the server rooms; the paper recommends this so liquid containers need not be carried into them.

Who should read it

Field and operations teams who receive, install and commission liquid-cooled racks, system integrators and facility engineers.

Document contents

1. Executive Summary
2. Terminology
3. Server Manufacturing level
4. Manufacturing
5. Rack Integration - from L1 to L10
6. Rack Integration - from L10 to L11
7. L10/L11 System Shipped without fluid
8. Transportation
9. Manifold preparation
10. Cold Plate preparation
11. Packaging for transport
12. Rack Integration
13. DC Integration
14. Delivery
15. Inspection
16. Preparation
17. Install, Commissioning, Operate, Decommission
18. Preconditions for installation
19. Installation
20. Commissioning process for cooling infrastructure
21. Operation
22. Decommission
23. Reuse Preparation
24. References

Related standards

• OCP Guidelines for Pre-Commission Preparation of TCS Row Manifolds
• OCP Leak Detection and Intervention white paper
• ASHRAE Liquid Cooling Guidelines

Read the document

Publication details

Authors

Gonzalez, M. (CEJN); Waechter, C.; Brill, T. (Cloud&Heat); Grau, J.; Marquis, T. (Parker); Archibald, M.; Stammer, S. (nVent); Turner, C. (CoolIT); Charest, G.; Fernandes, J. (Facebook); Gore, N. (Vertiv); Lee, C.; Chen, S. (Wiwynn)

Publisher

Open Compute Project Foundation

Year

2021

Document number

Rev 1.0

Document language

English

Source type

Industry body

Access

Open licence — downloadable

Licence

CC BY-SA 4.0 — share with attribution under the same licence

Licence evidence

www.opencompute.org

Citation