Quick Answer: Moving a GPU cluster from cloud to colocation means taking ownership of the physical hardware, the network fabric, and every operational layer the cloud provider was managing on your behalf. The process requires a structured audit of your cluster's performance baseline, power and cooling requirements, and interconnect architecture before any hardware ships. Done correctly, colocation can reduce total cost of ownership by 40-60% for sustained GPU workloads compared to cloud rates, while giving you full control over hardware selection and data handling.
Most AI teams arrive at the cloud-to-colocation decision the same way. The monthly GPU bill climbs faster than model performance does, and at some point someone in finance flags the discrepancy. The maths is straightforward enough. The execution is not.
What those conversations rarely surface is how much of the cloud platform's value lives below the compute API. Your GPU-to-GPU interconnect, your RDMA fabric, your subnet management, your checkpoint storage configuration: all of it has been handled for you. Moving to colocation does not just mean finding a building to put your servers in. It means taking ownership of an infrastructure stack that most cloud users have never had to touch.
This guide is written for the infrastructure leads and engineering teams who have made the decision and need to execute it without incident, starting with the pieces most migration playbooks skip over entirely.
Most teams underestimate how much work cloud providers absorb below the compute abstraction layer. The GPUs are visible and well-documented. The networking, power distribution, and operational support structures that make those GPUs useful at scale are largely invisible until you have to build them yourself.
When you run GPU workloads in the cloud, the interconnect fabric connecting your compute nodes is pre-built and managed by the provider. AWS uses Elastic Fabric Adapter (EFA), which provides RDMA-like performance over its internal network. Google Cloud provisions custom high-speed fabrics for GPU instances.
NVIDIA's Collective Communications Library (NCCL) detects and uses whatever fabric the cloud exposes, and your distributed training jobs communicate at high bandwidth without you configuring a single switch port or subnet route.
In colocation, that abstraction is completely absent. If your cluster runs InfiniBand, you need to deploy OpenSM as the subnet manager and validate that it is initialising and routing correctly before you add a single compute node.
If you opt for RoCEv2 instead, you need to configure Priority Flow Control and DCQCN congestion notification on every switch in the fabric to achieve lossless behaviour. The risk with RoCEv2 is well-documented: a misconfigured network will silently fall back to standard TCP, and your cluster will compute at a fraction of expected throughput with no obvious error to investigate.
The trade-off of managed networking is that you cannot tune it to your specific workload. Cloud providers choose fabric architectures that work across all tenants, not ones optimised for your particular training topology.
In colocation, rail-optimised InfiniBand architectures that align GPU pairs with NVLink domains are available and fully configurable. For clusters at scale, this is not a minor optimisation. Research on GPU cluster network topology design shows that network configuration errors account for roughly 10.7% of significant job failures in production GPU deployments, and topology choice directly influences how much of that risk you carry.
Cloud GPU pricing is not designed for teams running sustained, continuous workloads. It is designed for teams with variable demand who need infrastructure on short notice. For teams training models around the clock or running persistent inference pipelines, that flexibility premium compounds into a significant overpayment relative to owned hardware in a purpose-built facility.
The charge that catches most teams off-guard is not the GPU instance cost. It is data egress.
Moving training data from object storage into your compute environment, writing model checkpoints back out, and distributing inference outputs all generate outbound data transfer fees. AWS charges $0.09 per GB for the first 10TB of internet egress per month, and that baseline rate does not include NAT Gateway processing fees or cross-AZ transfer, which can push the effective per-GB cost to $0.135 or higher.
For teams moving petabyte-scale training datasets, egress alone can represent tens of thousands of dollars monthly. In colocation, that data moves within your own infrastructure at no per-byte cost.
The egress problem is especially visible in AI pipelines because the data volumes are large and the movement is continuous. Training data comes in. Checkpoints go out. Inference outputs move downstream. Each transfer is metered, and what looks like a manageable line item at small scale becomes a structural cost problem as model sizes grow.
Cloud GPU pricing for serious AI workloads is rarely truly on-demand. Teams commit to reserved instances to get predictable rates, and those reservations bill whether or not the GPUs are busy.
A 1,024-GPU H100 cluster on one-year reserved neocloud pricing runs roughly $13-18M annually, compared to $25-30M at standard hyperscaler on-demand rates. Owned hardware in colocation, at above 60% three-year utilisation, comes in substantially below either figure.
This is the calculation that has been driving the wave of cloud repatriation across enterprise AI teams. Once you can predict that your cluster will run above 60% utilisation over a three-year horizon, the total cost of ownership maths consistently favours colocation.
The failures in GPU cluster migrations rarely happen during the physical move. Hardware ships, gets racked, and powers on reliably. The problems surface when the colocation environment turns out to have insufficient per-rack power, when the interconnect delivers lower throughput than expected, or when no one has defined the operational model for managing a physical cluster before go-live. Every one of these problems is diagnosable before you sign a facility contract.
Before any hardware ships, benchmark your existing cloud cluster and record the results. NVIDIA NCCL Tests is the standard validation suite for GPU cluster communication performance, and the all_reduce_perf benchmark will measure inter-GPU bandwidth across your full node count, giving you a concrete performance target for your colocation deployment.
NCCL's automatic topology detection across PCIe, NVLink, InfiniBand, and RoCE lets you directly compare what the library sees in your cloud environment versus what it sees post-migration. If the colocation cluster does not match your cloud NCCL baseline, you have a configuration problem on your hands. That number is your go/no-go gate before any production workloads move.
A single NVIDIA H100 SXM GPU draws up to 700W at full load. A standard eight-GPU DGX H100 system consumes up to 19.8kW at the system level. At standard GPU cluster densities, you are routinely looking at 20-40kW per rack, which traditional colocation environments designed around 5-10kW racks cannot accommodate without major infrastructure upgrades. Before committing to any facility, verify the following:
Getting the physical floor plan and cabling topology signed off before hardware ships is not optional. Rack placement decisions made on-site at installation time are extremely difficult to undo.
This is the most consequential technical decision in a GPU cluster migration. InfiniBand NDR delivers 400Gbps per port with sub-microsecond latency and is lossless by design.
It is what NVIDIA's DGX SuperPOD reference architecture specifies and the default choice for large distributed training workloads. RoCEv2 runs RDMA semantics over standard Ethernet infrastructure, which reduces hardware cost substantially (InfiniBand switches run roughly three times the cost of comparable Ethernet), but it requires careful configuration of Priority Flow Control and DCQCN to achieve lossless behaviour.
For clusters above 256 GPUs running sustained training, InfiniBand is the safer choice. For smaller GPU server colocation deployments running inference rather than training, RoCEv2 is a practical and considerably less expensive alternative.
|
Interconnect |
Port Speed |
Latency |
Lossless |
Relative Switch Cost |
Best For |
|
InfiniBand NDR |
400Gbps |
Sub-1µs |
By design |
Baseline |
Large-scale distributed training |
|
RoCEv2 |
400GbE |
1-5µs |
Requires PFC/DCQCN config |
~3x lower |
Inference, smaller clusters |
|
Standard Ethernet |
Up to 400GbE |
Higher |
No |
Lowest |
Management traffic only |
A GPU cluster migration executed well is a structured programme with defined entry and exit criteria at each stage, not a single scheduled maintenance window. The teams that get through this without major disruption treat each phase as complete before moving to the next.
Facility readiness is not automatic. Pre-provision power circuits and verify that physical racks are installed and tested before hardware ships. Install InfiniBand switches and cabling infrastructure, run OpenSM, and confirm that the fabric initialises correctly before adding any GPU nodes.
Pre-order cross-connects through the facility's interconnection services so that upstream bandwidth is live well before you need it. Fibre provisioning at colocation facilities commonly takes two to four weeks from order placement. Starting this process late is the most common source of go-live delays across GPU migrations.
Active training jobs cannot survive a migration without a deliberate checkpointing strategy. A GPU training run interrupted without saving state has to restart from the beginning, and at the compute scales where colocation makes financial sense, that means days or weeks of wasted GPU time.
Before cutover, configure your training loop to save a complete checkpoint to durable storage, including model weights, optimiser state, learning rate schedule, and the current training step, then suspend the job cleanly. Good checkpoint hygiene at this stage also informs your ongoing backup and disaster recovery architecture in the new environment, since colocation requires you to own that continuity planning in a way that managed cloud storage services do not.
The sequence of the cutover matters as much as the timing. Bring up management networking first, then the InfiniBand or RoCEv2 fabric, then storage, then compute. Before loading any production workloads, validate each of the following:
Only after these checks pass should you route production traffic to the new environment and begin decommissioning cloud resources.
Acceptance testing is the gate between "the cluster runs" and "the cluster runs correctly." Run a representative training workload at full scale and compare throughput against your cloud baseline on step time, allreduce bandwidth, and GPU utilisation.
If any metric falls more than 5-10% below baseline, investigate before declaring go-live. The most common causes are fabric misconfiguration, NUMA topology misalignment between CPU and GPU memory access patterns, and thermal throttling if cooling capacity is undersized at the deployed rack density. Running acceptance tests on a real workload at full node count is the only reliable way to surface these issues before they affect production.
|
Phase |
Key Actions |
Exit Criteria |
|
Environment Prep |
Rack, cable, run OpenSM, provision cross-connects |
IB fabric validates, carrier circuits live |
|
Checkpoint and Suspend |
Save complete training state, suspend jobs cleanly |
Checkpoint confirmed in durable storage |
|
Cutover |
Bring up fabric layers, validate NCCL, confirm GPUDirect RDMA |
NCCL results match cloud baseline |
|
Acceptance Testing |
Full-scale workload, compare throughput vs pre-migration |
Within 5-10% of cloud performance baseline |
Moving GPU clusters to colocation changes where your hardware runs, but it also changes who is responsible for every layer of the stack below your application code. That shift is significantly larger than most engineering teams anticipate when they make the decision to repatriate their infrastructure.
In cloud environments, CUDA driver updates, GPU firmware patches, and kernel compatibility management are handled by the provider.
In colocation, your team owns all of it. CUDA drivers are not forward-compatible, meaning a driver version working with your current toolkit may not survive a framework upgrade. InfiniBand firmware updates need to be coordinated across all nodes simultaneously to avoid topology inconsistencies. OFED version mismatches between nodes can cause silent RDMA degradation that takes days to diagnose correctly.
Teams that treat GPU fleet management as an afterthought consistently run into unexplained throughput degradation months after a successful go-live.
The root cause is almost always a driver or firmware version drift between nodes that went undetected because no one owned the patching cycle. Planning your operational model before you commit to a facility contract is not optional. It is what separates a migration that stays clean over a three-year hardware lifespan from one that degrades quietly.
GPU fleet monitoring in colocation requires a purpose-built observability stack, not the general-purpose cloud monitoring dashboards you may have been relying on. A production GPU cluster should collect, at minimum:
Research on large-scale GPU cluster reliability has found that infrastructure-related failures, while affecting a small percentage of jobs overall, account for a disproportionate share of total GPU runtime loss.
The jobs most likely to fail are the large, long-running training runs that consume the most compute hours. Early detection of degraded hardware, thermal drift, or fabric issues is the only effective mitigation. Monitoring that fires an alert after a job fails is reactive. Monitoring that catches ECC error rate trends or link degradation before a failure is the operational target.
Moving AI GPU workloads from cloud to high-density GPU colocation is a sound infrastructure decision once workloads are predictable and sustained. What separates a clean migration from a costly one is the facility you land in. Not every colocation provider can support the power densities modern GPU clusters require, and not all of them will be transparent about that limitation during the evaluation process.
Qu Data Centres purpose-built its facilities to support high-density compute alongside the managed services teams that enterprise AI workloads require. Across our five Canadian markets, our facilities give teams the flexibility to deploy where they need to be, whether that is near financial services infrastructure in Toronto, government networks in Ottawa, or energy sector operations in Alberta.
With SOC 2, ISO 27001, HIPAA, and PCI DSS certifications across the portfolio, your compliance baseline is already in place. Our high-availability connectivity options provide redundant paths to the internet and to cloud platforms you may still use for overflow workloads. Because we are 100% Canadian-owned and operated, your data, model weights, and operations remain under Canadian legal jurisdiction, full stop.
Book a facility tour and bring your power density and rack count requirements. We will show you exactly what our facilities can support.
GPU colocation means you own the hardware and rent the physical space, power, and cooling from a data centre operator. Cloud GPU services rent you access to hardware the provider owns. Colocation gives you full control over hardware selection, firmware, and network configuration, but you are responsible for managing those layers rather than relying on the provider's abstraction.
The crossover point is generally above 60% GPU utilisation sustained over a three-year period. Below that threshold, cloud flexibility justifies the premium. Above it, owned hardware in colocation delivers lower total cost of ownership. Egress fees compound the case further for data-heavy AI pipelines that continuously move training data and model checkpoints.
Colocation rents physical space and power while you own and manage the hardware. Managed hosting means the provider also manages the hardware and software stack. GPU cloud services are fully managed compute rented on-demand with no hardware ownership. Colocation sits closest to on-premise in terms of control and cost structure, with the benefit of professional facility infrastructure you could not cost-effectively build yourself.
High-density GPU colocation refers to facilities capable of supporting rack power densities above 15-30kW per rack, the minimum threshold for modern GPU server deployments. Standard colocation is designed around 5-10kW per rack. High-density facilities include upgraded power distribution, liquid cooling support, reinforced flooring, and purpose-built cabling pathways for InfiniBand or high-speed Ethernet fabric.
Active training jobs must be checkpointed before migration begins. This means saving a complete copy of model weights, optimiser state, and training progress to durable storage. The cluster is migrated and validated in the new colocation environment, then training resumes from the saved checkpoint. Without this step, a migration-related interruption means restarting the training job from the beginning.