Best Practices Guide for Energy-Efficient Data Center Design (Revised July 2024)

Overview

The guide was written by Otto Van Geet and David Sickinger at NREL and supported by the U.S. Department of Energy's Federal Energy Management Program. It summarises best practices across information technology systems and their environmental conditions, air management, cooling and electrical systems, and heat recovery. IT and environmental conditions come first because measures there cascade into secondary savings in the mechanical and electrical systems. It does not claim to give the most efficient design; it offers suggestions that help across a wide range of scenarios.

The IT section notes that rack servers are the largest part of IT energy load and that average server utilisation in enterprise settings is 20% to 40%; server efficiency rises about 50% when processor utilisation goes from 20% to 30%. ENERGY STAR servers are on average 30% more efficient and are required for federal data centers. Power supply efficiency was historically 60% to 70% and can reach 92% with 80 PLUS Titanium; the best load range is 40% to 60%. Virtualisation, storage consolidation and thin provisioning reduce IT load and cooling size.

The environmental and air management sections give the 2021 ASHRAE recommended range of 18-27 °C for classes A1-A4, with allowable ranges of 15-32 °C for A1 and 10-35 °C for A2. Supply air should be controlled so the most demanding inlet stays close to 80 °F, with 77-79 °F a practical setpoint. Raising inlet temperature from 77 °F to 91 °F can raise server fan speed 30% and, by the fan affinity law, more than double fan power, though cooling savings usually exceed this. Cable management, blanking panels, hot/cold aisle separation and containment, and at least 24 inches of clear raised-floor height are recommended.

The cooling section covers chilled water at 55 °F or higher, a 5-7 °F condenser water approach, low pressure drop air delivery, air-side and water-side economizers, thermal storage and direct liquid cooling. Densities of 60 kW per rack were seen in 2013 and recent liquid-cooled racks exceed 125 kW. In the electrical section double-conversion UPS efficiency rose from 85-90% in the 1990s to 95% or more in 2023 and reaches 99% in eco mode; a dry-type transformer with a 176 °F rise uses 13-21% less energy than a 302 °F-rise unit. A 2007 LBNL study found 7% energy reduction with 380 V dc distribution.

The benchmarking section gives PUE values of 1.6 standard, 1.4 good and 1.1 better and cites a 2022 Uptime Institute average of 1.55 for large data centers. ERE subtracts reused energy and can fall below 1.0; WUE is annual water use divided by IT energy. Cooling system efficiency is 1.1 kW/ton standard, 0.8 good and 0.6 better. The guide also introduces the ISO/IEC 30134 series and monitoring with DCIM. It is a best-practice source prepared in a U.S. federal context, not a design standard, and product names do not imply endorsement.

Key points

1. The ASHRAE recommended air-cooling envelope (classes A1-A4) is 18-27 °C dry bulb; allowable ranges are 15-32 °C for A1 and 10-35 °C for A2.
2. Data hall supply air should be controlled so the most demanding inlet stays as close to 80 °F as possible; a setpoint of 77-79 °F allows a margin for error.
3. Chilled water at 55 °F or higher improves chiller efficiency and avoids unnecessary dehumidification; a condenser water approach of 5-7 °F is recommended.
4. Compute racks reached 60 kW in 2013, and recent liquid-cooled racks exceed 125 kW.
5. Double-conversion UPS efficiency rose from 85-90% in the 1990s to 95% or more in 2023; raising it from 90% to 95% for a 13,140 MWh/yr IT load saves about 768,421 kWh/yr.
6. UPS eco or energy-saver modes can reach up to 99% efficiency; a dry-type transformer with a 176 °F rise uses 13-21% less energy than a 302 °F-rise unit.
7. PUE is annual facility energy divided by annual IT energy; benchmarks are 1.6 standard, 1.4 good and 1.1 better, and a 2022 Uptime Institute average of 1.55 for large data centers is cited.
8. ERE subtracts reused energy from facility energy and can fall below 1.0; site WUE is annual site water use divided by IT energy in L/kWh.
9. Air-side economizer opportunity should be judged on local climate, humidity control and outdoor contamination, not temperature alone.
10. A water-side economizer can bypass chillers when its tower and heat exchanger can reach the chilled water supply setpoint, otherwise it reduces chiller load; an oversized open tower with a separate heat exchanger allows an approach below 3 °F.

Using it in design

• Weigh the supply air setpoint against server fan energy: raise it in small steps within the ASHRAE recommended range so the most demanding inlet nears 80 °F, only after air management is in place, and control to IT inlet temperature rather than return temperature.
• Write cable management, blanking of rack openings, hot/cold aisle separation and containment, and at least 24 inches of clear raised-floor height into the design requirements for air management.
• Design the chilled water system with 55 °F or higher supply, a 5-7 °F condenser water approach and chillers and pumps selected for part-load efficiency; assess air-side or water-side economizer hours for the local climate.
• When selecting UPS, account for the low load factor of N+1 layouts (for example two units at 30% load: raising load factor to 40% adds 1.2% efficiency), and include eco mode, transformer temperature rise and PDU loading (20-50%) in loss calculations.
• Evaluate direct liquid cooling early for high-density racks; ASHRAE defined Class H1 for high-density servers.
• For heat reuse, test the conditions: a nearby heat host, a temperature level that suits it (ideally without heat pumps), preferably common ownership and supporting policy; set PUE together with ERE, WUE and CUE as target metrics.

Who should read it

Mechanical and electrical engineers who design new or retrofitted data centers, energy managers and design teams who set benchmarking targets.

Document contents

1. Background: Key Steps to Sustainable Data Centers
2. Information Technology Systems (Cloud and Colocation, Efficient Servers, Storage, Network Equipment, Power Supplies, Consolidation)
3. Environmental Conditions: 2021 ASHRAE Equipment Thermal Guidelines
4. Air Management (Cable Management, Aisle Separation and Containment, Supply and Return Air, Raising Set Points)
5. Cooling Systems: Direct Expansion Systems
6. Cooling Systems: Air Handlers
7. High-Efficiency Chilled Water Systems
8. Free Cooling (Air-Side and Water-Side Economizers)
9. Thermal Storage
10. Direct Liquid Cooling
11. Humidification
12. Controls
13. Electrical Systems: Power Distribution (UPS, PDU, Distribution Voltage)
14. Demand Response
15. DC Power
16. Lighting
17. Use of Waste Heat for Energy-Efficient Design
18. Benchmarking (PUE, ERE, WUE, CUE, Airflow Efficiency, Cooling System Efficiency, ISO/IEC 30134, DCIM)
19. Conclusion

Read the document

Publication details

Authors

Otto Van Geet; David Sickinger

Publisher

U.S. DOE FEMP / NREL

Year

2024

Document language

English

Source type

Government / public

Access

Open licence — downloadable

Licence

U.S. public domain — U.S. Government work; free to reuse

Licence evidence

www.energy.gov

Citation