Calculating Your True GPU TCO: Cloud Rentals vs. BYOGPU Colocation
Let's be honest — GPU pricing is deliberately confusing. Cloud providers advertise hourly rates that look reasonable until you actually run the numbers for a production workload. Then you discover egress fees, storage costs, premium support charges, and a dozen other line items that weren't in the initial quote.
I've watched engineering teams get burned by this repeatedly. They budget for $5/hour GPU instances, then get hit with a $47,000 bill at month-end because nobody accounted for data transfer or the fact that training jobs don't pause themselves at 5 PM.
So let's do the math properly. We're gonna calculate the true total cost of ownership (TCO) for both cloud GPU rentals and BYOGPU colocation over 12 and 36 months, because that's how long you'll actually be running these workloads.
The Cloud GPU Math Everyone Gets Wrong
Here's a typical scenario: You need an NVIDIA H100 for training large language models. AWS charges about $32/hour for a p5.48xlarge instance with 8x H100s. Quick napkin math says that's $23,040 per month if you run it 24/7.
But that's not your real cost. Not even close.
Hidden Costs in Cloud GPU Rentals
1. Egress Fees (The Big One)
AWS charges $0.09/GB for data transfer out. If you're training models and need to move datasets, checkpoints, or inference results around, this adds up fast. A 500 GB model checkpoint downloaded 20 times during training? That's $900. Per checkpoint.
Typical ML workloads generate 5-15 TB of egress monthly. At $0.09/GB, that's $450-$1,350/month you probably didn't budget for.
2. Storage Costs
GPU instances don't include persistent storage. You'll need EBS volumes or S3:
- EBS gp3: $0.08/GB-month
- S3 Standard: $0.023/GB-month
- S3 Intelligent-Tiering: $0.0025-$0.023/GB-month
For a 10 TB training dataset on EBS, that's $800/month. On S3, it's $230/month minimum.
3. Premium Support
If you're running production workloads, you need Business or Enterprise support:
- Business Support: 10% of monthly spend (minimum $100/month)
- Enterprise Support: 10% of monthly spend up to $150K, then tiered
On a $25,000/month GPU bill, that's $2,500/month just for support.
4. Reserved Instance Gotchas
Yes, you can get discounts with 1-year or 3-year reserved instances. But GPU instance types change constantly. The H100 instances you reserve today might be obsolete when H200 or B200 instances launch. And you're locked in — no refunds, no exchanges.
5. Idle Time Costs
Unless you're running perfectly optimized jobs with zero downtime, you're paying for idle GPUs. Debugging a training script? That's billable hours. Waiting for data preprocessing? Billable. Model not converging and you need to rethink your architecture? Still billable.
Industry average GPU utilization is around 60-70%. That means 30-40% of your spend is literal waste.
Real Cloud GPU TCO Calculation
Let's price out a realistic scenario: 4x NVIDIA H100 GPUs running 24/7 for ML training.
AWS p5.12xlarge (4x H100 80GB)
- Instance cost: $98.32/hour × 730 hours = $71,774/month
- EBS storage (10 TB): $800/month
- Data transfer (8 TB/month): $720/month
- S3 storage (5 TB): $115/month
- Business Support (10%): $7,341/month
- Total monthly: $80,750
- 12-month TCO: $969,000
- 36-month TCO: $2,907,000
And that's assuming zero downtime, perfect utilization, and no unexpected costs. In reality, add another 15-20% buffer.
The BYOGPU Colocation Alternative
Now let's look at the same workload with BYOGPU colocation at Bit Refinery.
Upfront Hardware Costs
4x NVIDIA H100 80GB PCIe
- GPU cost: ~$30,000 each = $120,000
- Server chassis (Dell R760 or similar): $8,000
- Networking/cabling: $2,000
- Total hardware: $130,000
Yes, that's a big upfront number. But you own the hardware — it's a capital expense you can depreciate, and it has resale value.
Monthly Colocation Costs
Bit Refinery BYOGPU Pricing
- 4x GPU colocation: $600/GPU × 4 = $2,400/month
- Includes: rack space, power, cooling, 1 Gbps network, remote hands, monitoring
- Egress fees: $0 (unlimited bandwidth)
- Storage: Included in server (NVMe drives)
- Support: Included (24/7 monitoring, ticketing, live chat)
- Total monthly: $2,400
Total Cost of Ownership
12-month TCO:
- Hardware: $130,000
- Colocation (12 months): $28,800
- Total: $158,800
36-month TCO:
- Hardware: $130,000
- Colocation (36 months): $86,400
- Total: $216,400
The Breakeven Analysis
Here's where it gets interesting.
Cloud vs. BYOGPU savings:
- 12 months: $969,000 - $158,800 = $810,200 saved (84% reduction)
- 36 months: $2,907,000 - $216,400 = $2,690,600 saved (93% reduction)
The breakeven point is roughly 2 months. After 60 days, you've paid off the hardware and everything else is pure savings.
Even if you factor in a 30% resale value loss on the GPUs after 3 years ($36,000), you're still saving $2.65 million.
When Cloud GPU Rentals Make Sense
I'm not gonna pretend colocation is always the answer. Cloud rentals make sense when:
- Workloads are genuinely sporadic — you need GPUs for 2-3 days per month, not 24/7
- You're experimenting — trying different GPU types before committing to hardware
- You need instant scaling — going from 4 GPUs to 40 GPUs for a week-long training run
- Capital constraints — you can't front $130K for hardware purchases
But if you're running continuous training, inference serving, or any workload with predictable baseline demand, you're lighting money on fire with cloud rentals.
The Hybrid Approach: Own the Base, Rent the Spike
This is Bit Refinery's core philosophy, and it's how most sophisticated ML teams operate:
- Own dedicated GPUs for baseline workloads (training, development, small-scale inference)
- Rent cloud GPUs for spike demand (hyperparameter sweeps, large-scale inference bursts)
You get cost efficiency from owned hardware plus elasticity from cloud resources. Best of both worlds.
Hidden Benefits of BYOGPU Colocation
1. No Egress Fees
This is huge. Move 50 TB of training data around? Free. Download model checkpoints 100 times? Free. Stream inference results to your application? Free.
Unlimited 1 Gbps bandwidth is included. Need 10 Gbps? It's available at a flat monthly rate, not usage-based pricing.
2. Predictable Costs
Your bill is the same every month. No surprises, no usage spikes, no "we accidentally left an instance running" disasters. Finance teams love this.
3. Full Hardware Control
You get SSH, IPMI, and VPN access. Install whatever drivers, frameworks, or custom kernels you need. No virtualization overhead, no hypervisor tax.
4. Data Sovereignty
Your data stays on your hardware in a SOC 2 compliant facility. No shared tenancy, no noisy neighbors, no cloud provider accessing your training data.
5. Hardware Resale Value
GPUs hold value surprisingly well. Even 2-year-old H100s will have a resale market when you upgrade to whatever NVIDIA ships next.
Real-World Example: ML Startup Saves $1.2M/Year
One of our customers — a computer vision startup — was spending $140K/month on AWS p4d instances (8x A100 GPUs). They were burning through their Series A funding on cloud bills.
They bought 8x A100 GPUs for $80K and moved to BYOGPU colocation at $4,800/month. Their new monthly cost: $4,800 vs. $140,000.
Annual savings: $1,622,400. They paid off the hardware in 18 days.
They still use AWS for burst workloads, but their baseline infrastructure costs dropped 97%.
How to Calculate Your Own TCO
Here's a simple framework:
1. Calculate True Cloud Costs
- Instance hourly rate × 730 hours/month
- Add egress fees (estimate 5-15 TB/month)
- Add storage costs (EBS + S3)
- Add support costs (10% of spend)
- Multiply by 12 or 36 months
2. Calculate BYOGPU Costs
- GPU hardware cost (upfront)
- Server/chassis cost (upfront)
- Colocation fee × months ($600/GPU/month at Bit Refinery)
- Subtract estimated resale value after 3 years
3. Compare
- If cloud TCO > BYOGPU TCO by month 6, colocation wins
- If cloud TCO < BYOGPU TCO after 12 months, stick with cloud rentals
The Bottom Line
For continuous GPU workloads, cloud rentals are a 3-5x markup over owned hardware. The math isn't even close.
If you're running GPUs 24/7 for more than 2 months, you should own the hardware and colocate it. If you're running sporadic experiments, rent from the cloud.
And if you're doing both — which most ML teams are — use the hybrid model. Own the base, rent the spike.
Want to run the numbers for your specific workload? We've got a TCO calculator and can walk through your architecture. No sales pitch, just honest math.
Because at the end of the day, GPU infrastructure is too expensive to guess at. You need real numbers, not marketing fluff.