There's a persistent myth in infrastructure circles that you have to pick a side. Either you're a cloud-native shop spinning up H100s on AWS at $32/hour, or you're a bare metal purist who owns every rack and refuses to touch a hypervisor. The reality for most engineering teams is messier and more interesting than either camp wants to admit.
What actually works — and what we see over and over with teams doing serious AI/ML work — is a hybrid stack. Dedicated bare metal for your GPU workloads and persistent data, combined with virtual machines for the orchestration layer, dev environments, and auxiliary services. It's not glamorous. But it's genuinely flexible and it keeps your costs from spiraling.
Let me walk through why this combination makes sense and what it looks like in practice.
The Problem With Going All-In on Cloud GPUs
Cloud GPU pricing has gotten a little absurd. An NVIDIA H100 on AWS (p4d.24xlarge) runs somewhere north of $30/hour. Even if you're only running training jobs 12 hours a day, five days a week, that's over $7,000/month for a single GPU node. And that's before egress. Before storage. Before the other 15 services your pipeline touches.
The egress piece is the one that sneaks up on people. If you're doing anything with large datasets — and if you're training models, you are — moving data in and out of cloud storage adds up fast. We've seen teams hit $16,000+ in monthly egress charges alone. That's not a rounding error, that's a headcount decision. Many teams realize you're paying someone else's mortgage when relying solely on these rental models for long-term projects.
So teams start thinking about owning their GPUs. Which makes sense. But then they swing too far the other way.
The Problem With Going All-In on Bare Metal
Pure bare metal is powerful but inflexible. You've got a fixed number of physical machines, and when you need to spin up a quick dev environment, run a lightweight inference API, or isolate a team's workload from production, you're either overprovisioning or doing awkward manual work.
Bare metal also doesn't give you great isolation by default. If three data scientists are sharing a GPU server and one of them kicks off a memory-hungry job, everyone feels it. You need some kind of virtualization layer to carve things up sensibly — and that's not a weakness, it's just good infrastructure design.
What the Hybrid Stack Actually Looks Like
Here's the setup that tends to work well for teams doing a mix of model training, inference serving, and data engineering work:
Layer 1: Bare metal GPU servers for training and heavy inference
This is where you put your owned GPUs — H100s, A100s, RTX 4090s, whatever your workload calls for. These machines run bare metal because you want every bit of PCIe bandwidth, every GB of HBM, with no hypervisor tax. A training job that takes 6 hours on bare metal might take 8+ on a virtualized instance. That difference compounds.
With a BYOGPU colocation setup, you ship your hardware, it gets racked and connected within 48 hours, and you get full SSH and IPMI access. You're paying for the facility, power, and networking — not for the GPU itself at cloud markup. This transition from API tax to owned infrastructure is becoming the standard for mature ML teams.
Layer 2: Virtual machines for everything else
Dev environments, staging inference endpoints, data preprocessing pipelines, monitoring stacks, internal tooling — none of this needs bare metal. It needs isolation, easy provisioning, and the ability to resize without calling anyone.
A modern virtualization platform like VergeOS (which is what we use) handles this really well. It's an ultraconverged setup — compute, storage, and networking in one OS — and you get nested tenants, instant snapshots, and software-defined networking without the per-core licensing nonsense that made VMware so painful. You can give each team their own isolated environment, set resource quotas, and snapshot before anything risky.
Layer 3: S3-compatible object storage on bare metal
This one's easy to overlook but it matters a lot. Your training datasets, model checkpoints, and inference artifacts should live on dedicated MinIO storage — not in cloud object storage where every read and write has a cost attached. Flat monthly pricing, no egress fees, and you can size it to your actual data volume instead of paying per-GB indefinitely.
Why the Isolation Actually Matters
One thing that surprises teams when they move to this kind of setup: the VM layer isn't just about cost. It's about team dynamics.
When your ML engineers have their own isolated VergeOS tenant, they can experiment without worrying about breaking the production inference stack. They can snapshot their environment before a major dependency upgrade. They can spin up a copy of their dev setup for a contractor without giving that person access to anything sensitive. For organizations in highly regulated sectors, this level of control is why data sovereignty isn't optional anymore when building out AI capabilities.
This is stuff that sounds boring until you've had a 2am incident because someone ran pip install --upgrade on a shared server.
The Cost Picture
Let's be concrete. A team running two H100s for training and a modest VM cluster for auxiliary workloads might look something like this:
- 2x H100 GPUs colocated: ~$1,200/month
- Bare metal server (Gold tier, 80 cores, 1TB RAM, 44TB SSD): $2,800/month
- Unlimited bandwidth, no egress fees: included
Compare that to running equivalent cloud infrastructure — two H100 nodes plus storage plus egress for a data-heavy team — and you're easily looking at $15,000–$20,000/month, depending on utilization patterns. When you look at pricing what the cloud premium actually buys, the math for dedicated hardware becomes undeniable.
The savings are real, but honestly the more compelling argument for most teams is the predictability. Fixed monthly costs mean you can actually budget. No surprise bills at the end of the month because someone forgot to terminate a spot instance.
When to Actually Use Cloud GPUs
This isn't an anti-cloud post. There are legitimate reasons to burst into cloud GPU capacity — a one-time large training run, a product launch spike, experimentation with a new architecture before you commit to hardware. The "own the base, rent the spike" model makes sense here.
The mistake is treating cloud GPUs as your primary training infrastructure when you have consistent, predictable workloads. If you're running training jobs most days of the week, you're subsidizing AWS's margins in a pretty significant way.
Getting Started
The teams that do this well usually start simple: one bare metal server, a small VergeOS VM cluster for dev/staging, and MinIO for storage. They run it for a quarter, measure actual utilization, and then decide whether to expand the GPU capacity, add more VM resources, or both.
You don't need to architect the perfect system on day one. You need something that works, that you can actually operate, and that doesn't cost you $20k/month while you're still figuring out your model architecture.
If you're thinking through what this might look like for your team's specific workload, we're happy to talk through it. No sales pitch, just a conversation about what actually makes sense for your situation.