Cooling the Data Center: Design of a Mechanical Controls Owner Project Requirements (OPR) Template

Overview

The authors, from Montclair State University, Oak Ridge, Lawrence Livermore and Sandia national laboratories, Microsoft, JLL and Chilldyne, argue that mechanical cooling controls are often overlooked in pre-design. They are treated as second-tier, or the unique load profile of HPC systems is misunderstood. The consequences listed include reduced cooling efficiency, poor operator interfaces, higher operating costs, reduced reliability and resilience, no room for expansion, and controls redesigned on the fly during construction or after go-live. HPC systems can be refreshed on a cycle of about 3 years, so the cooling may be upgraded on a similar cycle.

The paper defines the OPR as a written document that details the functional requirements of a project and how it will be used and operated. It is written in pre-design, held by the owner, and feeds the architects' and engineers' Basis of Design, which is owned by them and must be informed by the OPR. It also gives the commissioning provider objective acceptance criteria. The primary stakeholders are operations and facility managers at HPC sites; the primary users are the architects and engineers who design the controls.

The template was built by the Cooling Controls Team of the EE HPC Working Group, which has about 800 members from 20 countries. The team met weekly through 2020-2021, started from an ASHRAE sample OPR and ASHRAE commissioning guidelines, produced a draft in spring 2021 and collected feedback through meetings and a survey. Respondents rated the structure as working well and were overwhelmingly positive about the template's value. The proposed eight sections are key owner requirements, project description and background, objectives, functional uses, budget, performance criteria, commissioning and service and maintenance.

For the key owner requirements the paper lists topics with example wording from previous projects. These include the owner holding the pen, commissioning support with 100% point-to-point testing by the controls contractor plus 10-20% random sampling by the commissioning agent, cybersecurity over the full service life, avoiding obsolete or beta products, alarm escalation, training, documentation, integration with BAS, DCIM and EPMS, vendor service response, alignment with site standards and open protocols such as Redfish and BACnet/SC. The discussion states that the template addresses brownfield upgrades rather than greenfield, is still in development, and does not address future architecture changes such as quantum or cloud computing.

Key points

1. HPC system refresh cycles can be about 3 years, so mechanical cooling upgrades may follow a similar cycle.
2. The EE HPC Working Group has roughly 800 members from 20 countries; the template was drafted by its Cooling Controls Team in 2020-2021.
3. The template proposes eight OPR sections: key owner requirements, project description, objectives, functional uses, budget, performance criteria, commissioning, and service and maintenance.
4. Suggested commissioning requirement: 100% point-to-point testing by the controls contractor with 10-20% random sampling verified by the commissioning agent, plus functional, load bank and integrated systems tests.
5. Open communication protocols are recommended for cooling controls, with Redfish and BACnet/SC named as examples.
6. Controls should integrate seamlessly with existing BAS, DCIM and EPMS, and share the look and feel of existing interfaces; one example scenario is pump energy data not reaching the DCIM after a new control system.
7. Serviceability examples include separating high and low control voltages so night-time troubleshooting does not need protective equipment, and mounting controllers and sensors where they are visible from floor level.
8. The template covers brownfield upgrades, is still in development, and the full document is expected through the EE HPC WG website.

Using it in design

• Use the eight-section outline as the skeleton for the controls part of an Owner's Project Requirements before the Basis of Design starts.
• Write commissioning into the controls scope from day one, including the 100% point-to-point and 10-20% sampled verification split and load bank and integrated systems tests.
• Require open protocols (Redfish, BACnet/SC) and defined integration with the BAS, DCIM and EPMS, and test that every new point, such as pump energy, actually reaches the monitoring system.
• Add serviceability rules to the requirements: separated control voltages, accessible and readable sensors, vendor service response times, and end-of-life checks on hardware and software.
• State the thermal envelope or water classification that the cooling system must meet, referencing owner or industry standards.

Who should read it

Owners, facility managers and controls designers planning a mechanical cooling controls upgrade or new HPC installation who need a structured way to capture requirements.

Document contents

1. Abstract
2. I. Introduction
3. II. Owner's Project Requirements (OPR)
4. III. Template Design Process
5. IV. Proposed Mechanical Cooling Controls OPR Template
6. Key Owner's Project Requirements topics and examples
7. V. Discussion
8. Acknowledgments
9. References

Publication details

Authors

Stefan A. Robila; David Grant; Chris DePrater; Vali Sorell; Terry L. Rodgers; David Martinez; Shlomo Novotny

Publisher

IEEE (Energy Efficient HPC Working Group paper; hosted by Lawrence Berkeley National Laboratory)

Year

2021

Document language

English

Source type

Scientific

Access

Citation only

DOI

10.1109/Cluster48925.2021.00085

Licence

Licence conditions not established — confirm the conditions with the publisher

Licence evidence

datacenters.lbl.gov

Citation