DCDM1: Lessons Learned from the World's Most Energy Efficient Data Center (NREL/PR-2C00-70433, March 12, 2018)

Overview

The deck walks through why and how NREL built its ESIF data center. Planning began in 2006, the team committed to direct liquid cooling based on HPC trends, aimed for racks above 60 kW, no mechanical chillers, and reuse of waste heat for office and laboratory space through chilled beams. Two temperatures governed the design: IT supply water that could be produced at 24 °C (75 °F) on the hottest day, matching ASHRAE class W2, and return water at 35 °C (95 °F) to heat the facility on the coldest day. The facility is a 182,000 ft2 LEED Platinum building with a 10,000 ft2 data center, 10 MW at full build-out and a PUE target of 1.06 or lower.

The slides give the rationale for liquid cooling in numbers. Liquid conduits need 250-1,000 times less space than air conduits for the same heat, and liquids take 10-20 times less energy to transport it. Liquid-to-liquid exchangers have closer approach temperatures than coils, raising economizer hours. Racks are fed 480 VAC directly at 60-80 kW, ten 100-kW racks take less space than fifty 20-kW air-cooled racks, and at least 95% of rack heat is captured directly in liquid. Design considerations listed include compatible metals and water chemistry, CDUs with flow control and filtration, hydronic redundancy and cooling systems in series so the most sensitive equipment gets the coolest water.

The metrics slides define PUE, ERE (which subtracts reused energy), WUE (site water per IT kWh in L/kWh) and WUE source, which adds water used at the power plant through an energy water intensity factor times PUE. The presentation then covers the thermosyphon cooler hybrid. A dry cooler placed upstream of the tower lets the system run wet when it is hot and dry when it is not. Fan speed depends on the water-to-energy cost equivalence ratio and the difference between entering water and dry bulb. Modelling used $0.07/kWh and $6.08/kgal; the cumulative savings slide uses $5.18/kgal.

Results for the first year of TSC operation (September 2016 to August 2017) were an hourly average IT load of 888 kW, PUE 1.034, ERE 0.929, WUE 0.7 L/kWh (1.42 with cooling towers only) and WUE source 5.4 L/kWh using 4.542 L/kWh for Colorado, with entering TSC water averaging 28.9 °C (84 °F). The conclusions say warm-water liquid cooling proved very energy efficient, the system modification was straightforward, savings were in line with modelling and the hybrid increased resiliency. Favorable applications have year-round load, loop temperatures high relative to ambient, high water rates or restrictions, and an owner seeking water resiliency. Being slides, the content is brief and relies on the companion reports for detail.

Key points

1. NREL's design basis was IT supply water of 24 °C (75 °F), ASHRAE class W2, and return water of 35 °C (95 °F) to heat the facility on the coldest day.
2. The ESIF data center has 10,000 ft2, 10 MW at full build-out, no mechanical cooling, and a PUE target of 1.06 or lower.
3. Liquid conduits need 250-1,000 times less space than air conduits for the same heat, and liquids require 10-20 times less energy to transport it.
4. Racks are supplied with 480 VAC at 60-80 kW per rack; at least 95% of rack heat load is captured directly to liquid.
5. Ten 100-kW racks take much less space than the equivalent fifty 20-kW air-cooled racks.
6. Rack power density guidance: air cooling suits about 1-5 kW per rack, while 5-80 kW calls for liquid options such as chilled doors, cold plates, fixed cold plates or immersion.
7. First year with the thermosyphon: IT load 888 kW, PUE 1.034, ERE 0.929, WUE 0.7 L/kWh against 1.42 L/kWh with towers only, WUE source 5.4 L/kWh.
8. Water-cooled design uses the lower wet-bulb design day and about 80% less airflow than air-cooled systems, but needs reliable low-cost water.
9. Favorable hybrid applications have 24/7 year-round load, loop temperatures high relative to ambient, high water rates, or a goal of water resiliency.
10. The hybrid avoided an outage by allowing a cooling tower loop repair on a cold day.

Using it in design

• Fix two design temperatures early, the coolest supply you can produce on the hottest day and the return temperature needed for heat reuse, and select IT equipment against them.
• Use the 250-1,000 times space and 10-20 times energy transport ratios when comparing liquid and air distribution options.
• Arrange cooling loops in series so the most sensitive equipment gets the coolest water, and plan CDU filtration, metals compatibility and pump and exchanger redundancy.
• Track PUE, ERE, WUE and WUE source together so that reuse and water effects are not hidden by PUE.
• Screen a hybrid wet-dry heat rejection concept using the favorable characteristics list: year-round load, warm loop, high water cost or restrictions.
• Check water-to-energy cost equivalence for the site, because it sets how much dry cooling is worth running.

Who should read it

Data center designers and owners looking for a short, practical summary of NREL's liquid cooling, heat reuse and water-saving lessons.

Document contents

1. NREL's Dual Computing Mission
2. Planning for a New Data Center
3. NREL Data Center
4. Cooling Efficiency
5. Energy Efficient Data Centers
6. Metrics
7. Liquid Cooling Considerations
8. Air- and Water-Cooled System Options
9. Opportunities for Hybrid Wet/Dry Solutions
10. Thermosyphon Cooler
11. System Modeling Program
12. Controls Allow for Saving Both Water and Operational Cost
13. Favorable Application Characteristics
14. Data Center Metrics
15. Project Partner Update: Sandia National Laboratories
16. Conclusions

Publication details

Authors

Otto Van Geet; David Sickinger; Thomas Carter

Publisher

NREL

Year

2018

Document language

English

Source type

Government / public

Access

Citation only

Licence

Licence conditions not established — confirm the conditions with the publisher

Licence evidence

www.nrel.gov

Citation