30 C Coolant - A Durable Roadmap for the Future, Revision 1.0 (Oct 2024)

Overview

The document is an OCP white paper by authors from Meta, Vertiv, Samsung, AMD, NVIDIA and Intel (1 October 2024, CC BY-SA 4.0), written because liquid cooling is reaching hyperscale for high-density AI hardware while data centres take very long to design, build and depreciate. Its core proposal is a minimum coolant temperature of 30 C for the TCS loop at the silicon, presented as a valuable operating point for both operators and silicon providers. The paper stresses that 30 C is not meant to limit technology variety (immersion, cold plates, thermal interface materials) and that some opportunities below 30 C will remain.

The case rests on hardware trends. CPU power was flat at around 100 W for a decade and then rose above 400 W in five years, GPU/ASIC power has risen by hundreds of watts almost every year, and published GPU power exceeds 1 kW. 2.5D (CoWoS) packaging creates thermal problems: different maximum junction temperatures (typically 105 C for the SOC; for HBM 85 C single-refresh and 95 C double-refresh in early generations, later 95 C and 105 C), different stack heights (SOC 450 micron, HBM 720 micron), buried components and heat crosstalk. HBM power doubled from HBM2 to HBM3 and HBM4 is expected to come close to doubling HBM3.

The energy analysis assumes an 80 MW IT load, sixteen 5 MW pods with 40 racks each at 125 kW per rack (80% liquid, 20% air), external air-cooled free-cooling chillers, CDUs and indoor air-cooled units, N+1 redundancy at cooling unit level, 80% IT load and dry mode without adiabatic or evaporative cooling. One location was chosen for eight of the nine ASHRAE 2021 climate zones (subarctic/arctic excluded), chillers were sized on ASHRAE's 20-year maximum ambient plus 4 C, and hourly weather data were averaged over ten years. The CDU approach is 6 C, so facility water (FWS) supply temperatures for S20, S30 and S40 are 14, 24 and 34 C. TCS delta-T is typically 10 C with 25% propylene glycol.

The results are clear: moving from S20 to S30 sharply cuts chiller energy, while S30 to S40 gives a similar gain only in the extremely hot, very hot and hot zones. A cited study found about 20% lower annual chiller power when fluid temperature rose from 21 C to 26 C and another 20% from 26 C to 30 C in most climate zones. Higher TCS temperature increases free-cooling time and flattens differences between regions: a data centre at TCS 20 C in the "cold" zone uses the same power as one at TCS 30 C in the "hot" zone. S20 and S30 barely affect current design; S40 pushes standard chiller compressors outside their normal range and strains standard indoor units above 50 C return air.

To sustain durability the paper proposes four areas: HBM silicon improvements (placing the hottest die near the cooling, noting microbump and adhesive layers at about 60 and 1 W/m.K against silicon's about 120 W/m.K), IT hardware-level cooling (parallel cold plates, long-term operation at 50-75 psig), better data centre heat exchangers (cutting the approach from 6 C to 4 C raises FWS temperatures to 16, 26 and 36 C) and silicon packaging innovation. Its stated limits are that the analysis is in dry mode only at N+1 and 80% load, and that near-100% free cooling at high TCS does not guarantee a chiller-less design is practical; capacity must be confirmed at full load and in failure scenarios.

Key points

1. The proposal is a minimum durable coolant temperature of 30 C for the TCS loop at the silicon, creating a long-lived interface value between operators and silicon providers.
2. Moving to a lower temperature is costly: pulling an existing TCS loop below its design range can need extra chiller capacity or a new chiller type, while a higher requirement is met by raising set points.
3. ASHRAE 2022 facility water classes are W17, W27, W32, W40, W45 and W+; the paper uses S20, S30 and S40 as TCS reference temperatures.
4. CPU power was flat at about 100 W for a decade and then exceeded 400 W in five years; published GPU power exceeds 1 kW; HBM power doubled from HBM2 to HBM3.
5. Model assumptions: 80 MW IT, 16 pods of 5 MW, 40 racks per pod at 125 kW (80% liquid, 20% air), 80% load, N+1, dry mode, CDU approach 6 C (FWS supply 14/24/34 C), delta-T 10 C with 25% PG; chillers sized for maximum ambient + 4 C.
6. A cited study found about 20% lower annual chiller power from 21 C to 26 C fluid and another 20% from 26 C to 30 C.
7. S20 to S30 gives a large drop in chiller energy; S30 to S40 gains much less except in extremely hot, very hot and hot climates.
8. S20 and S30 fit current designs; S40 pushes standard chiller compressors out of range and standard indoor units above 50 C return air, needing special products.
9. Typical CDU flow design is cited as 1.5 LPM per kW; CDU approach temperature is typically 4-8 C.
10. The IT cooling loop should be designed for long-term operation at 50-75 psig; cutting the approach from 6 C to 4 C lets FWS run 2 C warmer without changing TCS temperature.

Using it in design

• Default the liquid cooling design wizard to an S30 (30 C) TCS supply with a 6 C CDU approach (24 C FWS) and 1.5 LPM per kW.
• Show the effect of choosing S20 or S40 on annual chiller energy and free-cooling hours for the site's climate, reflecting that S20 to S30 gains a lot and S30 to S40 mainly in hot climates.
• Warn that S40 designs need chillers and indoor air units rated for high return temperatures (return air above 50 C).
• Use 50-75 psig as the TCS design pressure range in pipe and component selection.
• Offer lowering the CDU and coil approach (6 C to 4 C) as a separate option for raising FWS temperature; the gain is clearer in hot climates and at S30-S40.
• Near-100% free cooling hours does not by itself make a chiller-less design viable; verify cooling capacity separately at full load and in failure scenarios.

Who should read it

Cooling designers, operators and silicon or CDU manufacturers setting TCS and facility water temperature targets for liquid-cooled AI data centres.

Document contents

1. Introduction
2. Compliance with Open Compute Project Tenets
3. Component Power Trends
4. GPU and CPU Power Trends
5. Thermal Stack in Modern Silicon
6. HBM
7. Why Does Durability Matter?
8. From a Silicon Manufacturer's Point of View
9. From a Data Center Operators' Point of View
10. Why 30 C is best for Durability
11. 30 C or above for the coolant temperature at the silicon
12. Data Center Energy Efficiency
13. How do we Maintain Durability in the Future?
14. High Bandwidth Memory Silicon Improvements
15. Improving IT Hardware-level Cooling Solutions
16. Improved Data Center Heat Exchangers
17. Silicon packaging innovations
18. Conclusion

Related standards

• ASHRAE TC 9.9 Liquid Cooling Guidelines (W-classes, 2022)
• OCP L-L CDU Test Methodology
• OCP Thermal Boundaries in AI Infrastructure

Read the document

Publication details

Authors

Mills, S.; Hang, Y.; Fernandes, J. (Meta); Moscheni, A. (Vertiv); Kim, J. (Samsung); Artman, P. (AMD); Steinke, M. (NVIDIA); Malone, C. (Meta); Carte, C. (Intel)

Publisher

Open Compute Project Foundation

Year

2024

Document number

Rev 1.0

Document language

English

Source type

Industry body

Access

Open licence — downloadable

Licence

CC BY-SA 4.0 — share with attribution under the same licence

Licence evidence

www.opencompute.org

Citation