USE CASES

What people actually run on a private GPU

Not benchmarks. Production workloads, with the model named, the card named, and the latency measured.

Most writing about self-hosted inference argues from a spreadsheet. These are applications running on hardware in our racks right now. Each page says what the workload is, why it is not sitting on a per-token API, what it costs to serve, and — where the work is genuinely unfinished — what has not been proven yet.

The common thread

Four reasons a workload leaves the API

No workload moves to dedicated hardware for one reason alone, but in practice it is some combination of these four. Every case study below is an instance of at least two.

The data is not allowed to leave

Some inputs disqualify a third-party API before the pricing conversation starts. Private message content, patient records, unreleased designs, anything under a contractual no-disclosure clause. A promise that a vendor will not retain your prompts is a contract. A model on a card you rent alone is a fact.

The load is steady, not bursty

Per-token pricing is a good deal for spiky, low-volume work and a bad one for a job that runs the same shape of request all day. Once a workload has a floor, a dedicated card turns a variable bill into a fixed one — and the arithmetic usually crosses over sooner than people expect.

The model needs to already be warm

Serverless GPU platforms load weights on demand, so the first request after a quiet period pays for the load. When a model stays resident on hardware nobody else is scheduling onto, there is no cold start to design around and no queue you do not control.

The weights need to stay put

An open-weights model on your own hardware does not get deprecated, silently re-tuned, or repriced. For anything with a published evaluation or a regulated audit trail, reproducing last quarter's output matters more than access to the newest frontier model.

Case studies

The workloads

LIVE IN PRODUCTIONConsumer security software

Haven

Personal data security that never ships your messages to a vendor

Haven scans a person’s own cloud accounts and devices for sensitive data they have forgotten about — a tax return in a chat thread, a password sent in plain text. Almost all of that classification runs on the user’s own machine. The small fraction the local model cannot decide runs on a GPU in Bit Refinery’s rack.

Model

Gemma 4 31B, FP8

Serving

vLLM in a rootless container

Hardware

One 48 GB NVIDIA card

Read the Haven case study

DEMO LIVE · EVALUATION IN PROGRESSElectric utilities · vegetation management

LimbWatch

Vegetation hazard triage from a phone photo, on one GPU

Photograph a tree near a power line and get back a structured work ticket: severity score, hazard type, conductor type, species, tree health, and the pole tag number read off the tag. An open-weights vision model does the reading, served from a single card. The demo is live; the fine-tune and the published accuracy table are not done yet.

Model

Gemma 4 12B (Apache 2.0)

Serving

vLLM with guided decoding

Hardware

One RTX 6000 Ada 48 GB

Read the LimbWatch case study

More case studies land here as workloads move into production and their operators agree to be named.

The shape of it

One card, one resident model, a flat bill

Both workloads on this page have the same architecture, and it is an unglamorous one. A single 48 GB card. One open-weights model loaded once and left there. An inference server in front of it, TLS at the edge, and no scheduler deciding to evict anything. That is the entire trick.

48 GB

One card is enough for a 12B vision model or a 31B text model in FP8

Cold starts. Weights are resident, so the first request of the day costs the same as the thousandth

Flat

Monthly cost that does not move when usage doubles, and no egress metering on responses

Where does your workload sit?

If your inference load has a floor, or your inputs are the kind you would rather not hand to a vendor, the arithmetic is worth doing. We will do it with you, honestly, including the cases where an API is still the right answer.