Density choices for AI training are increasingly complex

Overview

The article argues that rack density for AI training clusters no longer has a single right answer. At the top end sits an NVIDIA NVL72 rack rated at 132 kW; at the bottom are configurations below 20 kW. The author weighs these options against performance, IT cost, facility upgrades, floor loading and time to deploy. Framing the question as a trade-off gives decision makers a structure for comparing the alternatives side by side.

For the highest-density option, floor load is 1.4 metric tons per rack. At full load the cold plates handle about 100 kW while air exhaust exceeds 25 kW, so even a liquid-cooled rack leaves a significant air-side load. One step down, 64-GPU racks operate within 80-90 kW and weigh about one metric ton, but about 50% more racks are needed for equal training performance.

Among the lower-density options, four 8-GPU servers per rack stay under 50 kW and air cooling can be viable with containment, in-row cooling or rear-door heat exchangers. Two server frames per rack come to about 26 kW, and power capping can bring that to 22 kW using 400 V/32 A circuits. These options present alternatives aimed at reducing the facility upgrades required and the time to deploy.

As a short analysis it offers no detailed cost model or measured data. Its central message is that the highest density is not automatically the best choice, and that stepping down in density can allow faster deployment in existing facilities. It is best read as a decision framework showing which variables to weigh together.

Key points

1. The NVIDIA NVL72 rack is rated at 132 kW with a floor load of 1.4 metric tons per rack.
2. At full load, cold plates handle about 100 kW while air exhaust exceeds 25 kW.
3. 64-GPU racks operate within 80-90 kW and weigh about one metric ton.
4. The 64-GPU option needs about 50% more racks for equal training performance.
5. Four 8-GPU servers per rack stay under 50 kW; air cooling is viable with containment, in-row cooling or rear-door heat exchangers.
6. Two server frames per rack are about 26 kW; power capping to 22 kW is cited with 400 V/32 A circuits.

Using it in design

• Define AI rack presets at about 130, 85, 50, 26 and under 20 kW, each with its liquid versus air cooling split.
• Run a floor-loading check for every preset; the article's values span about 1.0 to 1.4 tons per rack.
• Even for liquid-cooled high-density racks, include the air exhaust above 25 kW in the air-side cooling calculation.
• For placement in an existing facility, compare the extra racks and floor area a lower-density option needs against the cost of facility upgrades.

Who should read it

Design engineers and investors deciding on rack density, cooling method and facility upgrades for AI clusters.

Publication details

Authors

D. Bizo

Publisher

Uptime Institute Journal (Uptime Institute)

Year

2025

Document language

English

Source type

Industry body

Access

Free to read

Licence

Copyright retained; citation only — the publisher’s terms apply to the original document

Citation