AI workloads are fundamentally different from traditional enterprise applications. Training and inference at scale introduce sustained high-density compute, extreme east-west traffic, and unprecedented power and cooling demands. For many organizations, this is not an upgrade cycle - it is a structural redesign. 

This article serves as a starting point for designing and building AI-ready data centers. Think of it as a checklist, one that draws directly from IT pros working in real-world environments. In a recent roundtable conversation part of our Tech Unscripted series, four IT leaders and infrastructure experts discuss the challenges of designing AI-ready data centers. Use this practical guide to align strategic thinking with actionable steps, bridging leadership insights and operational readiness. 

 Watch our Tech Unscripted discussion with infrastructure leaders on building AI-ready data centers that can handle high-density compute, low-latency networking, and future-proofed power and cooling requirements. 

How To Design and Build AI-Ready Data Centers: A Checklist 

A data center that is truly AI-ready must be able to support high-density compute, low-latency networking, and sustained power and cooling demands - all requirements for modern AI workloads. This checklist outlines the core infrastructure considerations required to AI-proof a data center, focusing on network design, operational intelligence, and systems-level readiness. It isn't easy, of course, but with the right strategy, you'll be ready for AI today and in the future.

1. Design the Network for GPU-to-GPU Communication, Not Just Throughput

This model is fundamentally different. Here's how it works: AI training and inference performance is often constrained by data movement, not raw compute. In practical terms, this means confirming that your network design supports the following: 

• High-throughput, low-latency east-west traffic between GPUs 
• Non-blocking bandwidth across large GPU clusters 
• Predictable performance at scale, not just peak speeds

There are several important factors to consider when designing. First, traditional TCP/IP stacks may introduce unacceptable overhead for large-scale GPU clusters. Then, specialized architectures - for example, low-latency Ethernet with RDMA/RoCE or HPC interconnects - are often required. And, when hundreds of GPUs operate in parallel, network topology matters just as much as link speed. 

2. Validate Network Performance Using Tail Metrics, Not Averages 

AI workloads are sensitive to the slowest component in the system. Your performance validation strategy should include: 99th percentile (tail) latency measurements, jitter analysis across GPU clusters, and congestion detection under sustained load, not burst testing. At a minimum, ensure the ability to: 

• Measure tail latency, not just mean throughput. 
• Identify GPU-level bottlenecks caused by network congestion. 
• Test performance during long-running training or inference cycles.

3. Plan for Next-Generation Network Capacity Early

AI infrastructure lifecycles are shortening as accelerator and interconnect technologies evolve rapidly. Consider these angles for future-proofing: 

• Emerging GPU platforms may require 800 Gbps Ethernet connectivity. 
• Higher-bandwidth links can reduce training time and lower TCO (total cost of ownership) for large models. 
• Capacity planning should assume faster generational turnover than traditional data center upgrades.

4. Treat Observability as a First-Class Infrastructure Requirement

Simple monitoring is insufficient for AI environments. AI-ready observability for large AI environments must handle millions of telemetry data points per second, multi-dimensional metrics across GPUs, servers, networks, and cooling systems, and the real-time correlation between performance, security, and infrastructure health.  

At a minimum, this requires the ability to: 

• Collect fine-grained telemetry from compute, network, and environmental systems. 
• Correlate performance data with real-time workload behavior. 
• Detect subtle anomalies before they impact model training or inference.

5. Enable Closed-Loop Automation for Network and Infrastructure Operations

Manual intervention does not scale in AI environments. An AI-ready data center should support automated responses to network, power, and thermal conditions in real time to maintain performance and SLAs. 

In practice, this includes rerouting traffic away from congested high-bandwidth links, reducing power draw in response to pre-failure thermal indicators, and enforcing security or performance policies without human intervention.

6. Integrate Security into the Data Path, Not Around It

AI workloads expand the attack surface across data, models, and infrastructure. At the infrastructure level, security considerations should include, the continuous validation of connection requests, detection of lateral movement within GPU clusters, and ongoing monitoring for unauthorized data transfers or policy violations. 

To achieve this, follow these best practices:  

• Treat every connection as untrusted by default. 
• Enforce identity- and application-specific access policies. 
• Monitor AI workloads independently rather than relying on coarse network boundaries. 

7. Account for Power Density at the Rack Level

AI accelerators dramatically change power consumption patterns, so your planning parameters will change significantly. Baseline planning assumptions are: 

• Traditional CPU racks: ~5-10 kW 
• GPU-accelerated racks: ~30-50 kW 
• Large AI systems: 80+ kW per rack 

To best account for this power density, you should redesign power distribution for sustained high-density loads, plan for frequent and significant power spikes, and protect against outages where downtime costs exceed traditional workloads.

8. Treat Cooling as a Strategic Constraint, Not an Afterthought

Cooling is often the limiting factor in AI scalability. In fact, a significant portion of AI energy consumption is tied to cooling, not compute. The reality is that air cooling is typically efficient only up to ~10-20 kW per rack. Beyond ~35 kW, air cooling becomes inefficient and unsustainable.  

Cooling is not a set and forget activity. Spend time evaluating alternative cooling strategies that make sense for your environment, such as:  

• Direct-to-chip liquid cooling for high-density accelerators 
• Rear-door heat exchangers for incremental upgrades 
• Immersion cooling for extreme future-proofing scenarios

9. Design for Energy Efficiency and Sustainability

The energy resources required to power AI data centers is beyond anything we've seen. Ineed, AI data centers can consume energy at city-scale levels. That takes a lot of planning, so you'll need to:  

• Optimize cooling efficiency alongside compute performance. 
• Reduce waste heat and energy loss at the system level. 
• Treat sustainability as a design constraint, not a reporting metric.

10. Align Infrastructure Strategy with an OpEx-Friendly Model

AI economics are unpredictable, as we've seen over the last year. From a business perspective, there's several reasons for this: AI hardware evolves faster than traditional depreciation cycles. Specialized talent and accelerator availability remain constrained. Fortunately, flexible consumption models can reduce long-term risk. To align with an OpEx-friendly model: 

• Avoid over-committing to fixed architectures. 
• Design modular systems that can evolve with AI workloads. 
• Balance performance gains against long-term operational cost. 

Design with Intention and Commit to Long-Term Architecture Requirements 

An AI-ready data center is defined by two tightly coupled objectives: 

• A high-performance, lossless network fabric capable of sustaining GPU-to-GPU communication at scale 
• A systems-level design that can support extreme power, cooling, observability, and automation requirements over time 

AI readiness is not a single upgrade. It is an ongoing architectural commitment - one that must be designed into the data center from the ground up. 

To learn more about how real organizations are tackling the Future of Work, from AI to remote access, check out our entire Tech Unscripted interview series: click to listen or watch this episode now.  

Cisco Champion Radio · Tech Unscripted: AI for Data Center