Liquid Cooling Commissioning: Lessons Learned @LRZ (EE HPC WG session, SC13)

Overview

This document is a short LRZ presentation given in the EE HPC WG workshop session at SC13 (ACM/IEEE Supercomputing 2013, Denver). Its subject is commissioning the warm-water cooling plant for SuperMUC, an IBM iDataPlex system with direct water cooling. The deck shows LRZ's four temperature-level cooling layout: warm water 30/36 C, cold water 14/20 C, very cold water 4/10 C, plus a glycol/heating loop.

In the first case, unexplained make-up water loss was traced to an incorrectly sized cooling-tower level sensor. The cooling-tower loop has a fill volume of 7 m3 with typical feed above 1 m3/h, and losses of 1-2 m3 over hours to days were observed. The root cause was that the main contractor fitted a level sensor of the wrong dimensions rather than the one recommended by the cooling-tower manufacturer.

The second case was an operating-control test: a -20 K inlet temperature step was applied on one cluster loop, and the response and COP (cooling capacity over power input) of the warm-water plant were tracked. The lesson is that sensors and controls specified by equipment makers must be enforced through commissioning. On the closing slide, LRZ frames efficiency with a four-pillar model: building infrastructure (PUE), system hardware, system software and applications. As a short presentation it contains no detailed measurement data; only the record's summary and key facts were used here.

Key points

1. LRZ runs four loops: warm water 30/36 C, cold water 14/20 C, very cold water 4/10 C, plus a glycol/heating loop.
2. Cooling-tower loop fill volume is 7 m3 with typical feed above 1 m3/h; losses of 1-2 m3 over hours to days were observed.
3. Root cause: the main contractor fitted a level sensor of the wrong dimensions rather than the one recommended by the cooling-tower manufacturer.
4. The operating-control test applied a -20 K inlet temperature step on one cluster loop and tracked the warm-water plant response and COP.
5. LRZ frames efficiency with four pillars: building infrastructure (PUE), system hardware, system software and applications.

Using it in design

• In commissioning checklists the tool generates for liquid-cooled plants, include verification that instrumentation matches manufacturer specifications.
• Include step-response tests of supply-temperature control in the checklist.
• Make make-up water metering a required point on cooling-tower loops.

Who should read it

Engineers who commission, operate or specify liquid-cooled plants.

Related standards

• ASHRAE Guideline 0
• ASHRAE Liquid Cooling Guidelines

Publication details

Authors

Detlef Labrenz

Publisher

Energy Efficient HPC Working Group

Year

2013

Document number

SC13 workshop presentation wp135s1

Event

SC13 (ACM/IEEE Supercomputing 2013), Denver - EE HPC WG workshop

Document language

English

Source type

Industry body

Access

Free to read

Licence

Copyright retained; citation only — the publisher’s terms apply to the original document

Citation