TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
arXiv preprint (Microsoft Azure Research / University of Illinois) · Jovan Stojkovic, Chaojie Zhang et al.
An arXiv preprint from Microsoft Azure Research and the University of Illinois derives thermal and power models from Azure GPU datacenter production telemetry and uses them to place GPU VMs within cooling and power limits.
On this page
Overview
The work is an arXiv preprint by Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Haoran Qiu, Rodrigo Fonseca, Josep Torrellas and Ricardo Bianchini (arXiv:2501.02600, 2025; Microsoft Azure Research / University of Illinois). Using months of production telemetry from Azure GPU datacenters, it shows that server inlet and GPU temperatures vary with outside temperature, row position and server layout, and that LLM inference configuration (model size, quantisation, parallelism, frequency, batch size) trades performance against temperature, power and quality. GPU temperature must stay below a threshold (for example 85 C) beyond which hardware throttles computation.
With outside-air or adiabatic cooling, inlet temperature is held (for example over 18 C) when cold outside and rises roughly linearly once outside temperature exceeds about 15 C; at 35 C outside, low versus high load changes inlet by about 2 C. Across over 3,000 GPUs at high load with comparable inlet temperatures in one datacenter there is a range of over 20 C, and some rows run up to 1-2 C warmer.
TAPAS uses these thermal and power models to place GPU VMs within cooling and power limits, route inference requests among SaaS instances, and reconfigure instances during load spikes or cooling and power emergencies. It maintained P99 tail latency while reducing maximum temperature by 17% and peak row power by 23%, enabling up to 40% additional capacity through oversubscription. In trace-driven simulation of hundreds of production racks, it reduced thermal and power throttling events by 97% and 99%.
Key points
- TAPAS maintained P99 tail latency while reducing maximum temperature by 17% and peak row power by 23%, enabling up to 40% additional capacity through oversubscription.
- In trace-driven simulation of hundreds of production racks, thermal and power throttling events fell by 97% and 99% against other practical policies.
- GPU temperature must stay below a threshold (for example 85 C) beyond which hardware throttles computation.
- With outside-air or adiabatic cooling, inlet temperature is held (over 18 C) when cold and rises roughly linearly above about 15 C outside; at 35 C outside, low versus high load changes inlet by about 2 C.
- Across over 3,000 GPUs at high load with comparable inlet temperatures in one datacenter there is a range of over 20 C, and some rows run 1-2 C warmer.
Using it in design
- Couple thermal and power headroom in the design tool: show how inlet temperature rise with outdoor conditions (for economiser or adiabatic systems) erodes GPU thermal margin and can trigger throttling.
- Allow a row-level power budget with an optional oversubscription factor tied to workload controls.
- Flag rows and positions prone to recirculation as hotspots in CFD and airflow summaries.
- Recommend placing the most thermally sensitive GPU loads in the coolest zones.
Monthly newsletter · free
Keep your reading list current
Once a month: new library sources, changes to Turkish regulations and upcoming data centre events.
SoSo CAD
Check cooling capacity alongside your design
Review zone heat loads, available capacity and the capacity balance after the largest unit is lost in SoSo CAD.
Request a demo
Who should read it
Engineers designing cooling and power for AI clusters, and cloud scheduling researchers.
Publication details
- Authors
- Jovan Stojkovic; Chaojie Zhang; Íñigo Goiri; Esha Choukse; Haoran Qiu; Rodrigo Fonseca; Josep Torrellas; Ricardo Bianchini
- Publisher
- arXiv preprint (Microsoft Azure Research / University of Illinois)
- Year
- 2025
- Document number
- arXiv:2501.02600
- Document language
- English
- Source type
- Scientific
- Access
- Free to read
- Licence
- CC BY-NC-SA 4.0 — share with attribution for noncommercial purposes under the same licence
Original licence note
arXiv CC BY-NC-SA 4.0 (non-commercial) - citation only
Citation
Jovan Stojkovic; Chaojie Zhang; Íñigo Goiri; Esha Choukse; Haoran Qiu; Rodrigo Fonseca; Josep Torrellas; Ricardo Bianchini (2025). TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms. arXiv preprint (Microsoft Azure Research / University of Illinois). arXiv:2501.02600. https://doi.org/10.48550/arXiv.2501.02600
Events on this subject
Browse the events calendarAll content and rights belong to the respective authors and publishers. SoSo CAD provides only the citation, its own summary and a link to the source, and is not responsible for the sources' content.
Related sources
All sources on this topicSchneider Electric 2026
How 6 AI Attributes Change Data Center Design
Add an AI cluster design profile with rack defaults of 750x1,200 mm, 48U, static 2,270 kg and dynamic 1,600 kg.
- GPU TDP trend: V100 300 W, A100 400 W, H100 700 W, B200 1,000 W, B300 1,400 W.
Schneider Electric 2024
Power and Cooling Capacity Management for Data Centers
For every rack, row and upstream device store design maximum supply, design maximum demand (nameplate or as-configured) and measured actual where available; enforce supply >= demand at each hierarchy level including under N-1.
- Safety margins are typically 10-20%, up to 30% or more with weak change control.
Vertiv 2023
Deploying Liquid Cooling in the Data Center: A Guide to High-Density Cooling (1 MW IT load)
Calculation default: DLC flow = 1.5 L/min per kW of liquid-captured heat; to adjust by dT use flow = P/(4.18 x dT) L/s.
- Newest AI servers approach 6-10 kW TDP per server
arXiv preprint (Microsoft Azure) 2023
POLCA: Power Oversubscription in LLM Cloud Providers
In the power planner, apply different diversity and oversubscription factors by workload: inference rows can be provisioned below sum-of-nameplate (for example peak about 80% of provisioned), while training rows should assume near-coincident peaks with only about 10% headroom.
- POLCA allows 30% more servers in the same inference GPU cluster while maintaining SLOs.
Open Compute Project Foundation 2020
Colocation Facility Guidelines for Deployment of Open Racks, Revision 4.0 (15 December 2020)
Check the delivery route against rack height plus 0.5 m, door and corridor widths, and the lift capacity, before agreeing to host or design for Open Racks.
- Maximum rack density is at least 8 kW (Acceptable) or 12 kW (Optimum); cold aisle width is at least 1,200 mm or 1,500 mm.
Headquarters, Department of the Army (U.S. Army Corps of Engineers) 2007
TM 5-698-1 Reliability/Availability of Electrical & Mechanical Systems for Command, Control, Communications, Computer, Intelligence, Surveillance and Reconnaissance (C4ISR) Facilities (19 January 2007)
When writing an availability target into a data centre specification, state the measurement interval as well; the same downtime gives 95.83 or 99.86 percent depending on the interval.
- With exponentially distributed failures, reliability at an operating time equal to the MTBF is only about 37 percent (e^-1 = 0.368).