Lawrence Krubner put a question on LinkedIn recently that deserves a longer answer than a comment box allows. He was quoted about $60,000 a month by Azure for the hardware needed to run NVIDIA's open weight models, and about $20,000 a month by vendors leasing him comparable machines. His question was not "why is cloud expensive." It was narrower and better: what does the extra $40,000 actually buy, and is "reliability and redundancy" a sufficient answer?
The comment thread filled with the usual defenses, and several of them are good. Rather than argue the headline number (we have done that already), here is an attempt to price each defense one at a time, so you can check which ones you are actually buying.
Short version of where this goes:
- The gap is real and the arithmetic checks out.
- Two of the five common defenses hold up under scrutiny.
- Egress is usually the line item that decides the outcome, and almost nobody models it.
- There is a utilization number below which the hyperscaler genuinely wins.
The gap, in numbers
Azure's ND96isr H100 v5 (eight H100 SXM5 cards, 80 GB each) lists at roughly $98 per hour on demand, which is about $71,000 a month if you leave it running. A one year reserved commitment brings it to roughly $63 per hour, or about $46,000 a month. So $60,000 sits right in the middle of that band, and Krubner's quote was not an outlier.
Dedicated GPU providers in 2026 are renting the same silicon at roughly $2.50 to $4.00 per GPU hour. Eight cards at $3.00 for a full month is about $17,500. That is his $20,000. The two quotes are describing the same hardware, and the ratio is roughly three to one.
Worth noting for anyone sizing this: the model class in question no longer demands eight cards. NVIDIA's Nemotron 3 Super at 120B parameters fits on two H100s at FP8, and on a single card at NVFP4. The Nano tier at 30B runs on one H200. If your quote is built around an eight card node because that is the SKU the hyperscaler sells, the first saving available to you has nothing to do with the vendor.
Defense 1: "You are paying for reliability and redundancy"
This is the one Krubner named, and it is the weakest.
An SLA (service level agreement) is not an insurance policy. It is a discount schedule. Azure's compute SLA pays service credits against your bill for that service, typically 10% to 25% of the monthly charge for the affected resource, and every major provider caps liability at roughly what you paid them. If a nine hour outage costs your business $200,000 in lost work, the SLA pays you a few thousand dollars off next month's invoice. That is not redundancy you bought. It is a partial refund.
Actual redundancy is a thing you build, and you can build it anywhere. Two machines in two facilities, with a tested failover, is the same design pattern whether the machines are in Azure or in a colocation cage. The commenter who pointed out that you can get true multi site redundancy for 25% to 35% of hyperscaler cost is describing arithmetic, not ideology: buy two of the cheap thing and you are still well under one of the expensive thing.
The honest version of this defense is narrower. You are paying for the provider's operational track record and for not having to design the failover yourself. That has value. It is not $40,000 a month of value for a single node running inference.
Defense 2: "You are paying for elasticity"
This one is real, and it is the reason the hyperscalers exist. It is also the one most often claimed by people who are not using it.
Elasticity is worth paying for in exact proportion to how much of the time your machines are off. The break even is simple. Take the monthly cost of a dedicated card, divide by the on demand hourly rate for the equivalent card, and you get the hours per month at which the two are the same price.
| Dedicated cost per card | On demand rate | Break even | As % of the month |
|---|---|---|---|
| $2,200 / mo | $12.29 / hr | 179 hours | 25% |
| $2,200 / mo | $7.93 / hr (1 yr reserved) | 277 hours | 38% |
| $790 / mo (48 GB class) | $2.50 / hr | 316 hours | 43% |
Above those lines, dedicated wins. Below them, on demand wins, and it wins decisively at the low end (a research team that runs a four hour job twice a week should not lease a rack).
So the question to ask yourself is not philosophical. Pull your last ninety days of GPU utilization. If your cards are busy more than about a third of the time, you are paying an elasticity premium for elasticity you are not consuming. Inference for a production application is usually a floor, not a spike, which is why this defense tends to collapse precisely for the workload Krubner is describing.
Defense 3: "You are avoiding a capital purchase"
True, and worth something, especially for a startup that would rather put its cash into people. One commenter made this point well: cloud lets you skip the up front investment, and that matters most when you are young and uncertain.
But it is a false binary. Between "rent by the hour from Azure" and "buy $700,000 of servers" sits a large middle: leasing dedicated hardware on a monthly term, or colocating machines you already own. Both are operating expense. Neither requires a purchase order to the CFO.
The reference case here is 37signals, which several commenters raised. Their published numbers: about $700,000 of Dell servers replaced roughly $2 million a year of cloud compute, and a separate $1.5 million of storage arrays replaced an S3 bill of about $1.3 million a year, with ongoing operating cost near $200,000. Their projected five year saving is over $10 million. That is a purchase, and it is not a template most companies should copy directly. The useful part is the ratio, not the method. When the payback period on hardware is under a year, renting that same hardware monthly from someone else still leaves you far ahead of the hourly price.
Defense 4: "You are paying for the staff you don't have to hire"
This is the strongest objection in the thread, and it came from Austin Oehlerking, who listed what the comparison usually omits: incident response, networking, staffing, buildout, and data integrity. He is right that redundancy alone is a thin accounting.
The question is which of those you actually offload. If you run virtual machines on Azure, you are not offloading incident response for your application, your model server, your CUDA versions, or your data pipeline. You are offloading facilities, hardware replacement, and the physical network. Those are real and they are worth real money, but they are also exactly what a managed colocation or managed dedicated provider does, at a fraction of the price, because they are selling you that work rather than bundling it into a per hour rate that assumes you might also want two hundred other services.
The genuine staffing argument applies to the layer above: if your team is leaning hard on managed databases, managed Kubernetes, managed identity, managed queues, and you would otherwise hire two engineers to run those, then the premium is buying something. Count how many of those you use. In most environments we have seen, teams running LLM inference are using three or four managed services, not thirty, and are paying the price of a platform they are consuming a tenth of.
Defense 5: The one nobody puts in the comparison
Egress.
AWS charges roughly $0.09 per GB out to the internet after the first 100 GB. Azure is roughly $0.087. At 200 TB a month, which is unremarkable for an application serving model responses and syncing checkpoints, that is about $17,400 a month. On the numbers above, the egress line alone is roughly the size of the entire $20,000 quote from the other vendor.
Egress does two things at once. It inflates the monthly bill, and it makes leaving expensive, which is the actual mechanism of lock in. When 37signals finally exited, AWS waived $250,000 in egress fees to let them move 18 petabytes out. Most companies do not get that phone call.
So when you compare two quotes, put the transfer volume in the model. It frequently decides the outcome on its own, and it is the number most likely to be missing from the slide your vendor showed you.
An afternoon's worth of homework
You can settle this for your own environment in about half a day:
- Utilization. What percentage of the last ninety days were your GPUs actually busy? Compare it to the break even table above.
- Egress. How many terabytes a month leave your account? Multiply by $0.087. Do it before you look at compute.
- Managed services. List the ones you genuinely depend on. If the list is short, you are paying platform rates for a virtual machine.
- Your SLA. How much did it pay you last year? Compare that to what an outage actually cost you.
- Model footprint. Confirm the card count you need at the quantization you will actually run, rather than the SKU the vendor sells.
If the answers come back "busy most of the time, heavy egress, four managed services," the $40,000 gap is not buying you anything you are using. If they come back "bursty, light egress, deep in the platform," the hyperscaler is the right answer and you should stop reading cost threads.
Where the hyperscalers still win
To be fair to the other side, there are cases where the premium is justified and we say so to prospects regularly: genuinely unpredictable demand, short experiments, workloads that need to sit next to a large dataset already in that provider's storage, teams with no operational capacity at all, and anything where a procurement requirement names the provider. Those are real. They are just narrower than the default assumption, and the default assumption is what a three to one price gap is built on.
For anyone working through this: we run dedicated GPU pods and bare metal with fixed monthly pricing and no egress charges, and we are happy to build the comparison spreadsheet with you even if the answer is that you should stay where you are. The arithmetic is the same either way.