---
title: "Own the Base, Route the Spike: Where Private GPUs Fit in a Model-Routing World"
url: "https://bitrefinery.com/blog/own-the-base-route-the-spike-where-private-gpus-fit-in-a-model-routing-world"
description: "Agent harnesses are becoming the control plane, and model routing is now a policy decision. Here's the case for making a self-hosted, flat-rate GPU endpoint the destination for your routed-down inference tier — with the cost math and a working config."
author: "Mike George, VP of Operations"
date: "2026-07-11"
lastmod: "2026-07-11"
tags: ["AI infrastructure", "model routing", "private LLM hosting", "GPU economics", "agents"]
source: "blog CMS"
---

# Own the Base, Route the Spike: Where Private GPUs Fit in a Model-Routing World

I run infrastructure for a living, so I read most AI announcements through one lens: what does this do to the bill, and who ends up owning the hardware? A pattern has been quietly showing up in agent tooling over the last few months that answers both questions in a way that's good for anyone building on top of open models — and I want to walk through it plainly, because I think a lot of teams are about to leave money and control on the table by not noticing it.

The pattern is harness-level model routing. The agent harness — the thing that actually runs your agent loop, calls tools, and manages context — is becoming the control plane. And one of the decisions it now makes, per task, is which model handles this step. Not "which model runs my whole app." Which model runs this call.

That's a bigger deal than it sounds, and it's where a flat-rate private GPU stops being a nice-to-have and starts being the obvious home for a large slice of your inference.

## The harness is the new control plane

If you're building agents, you've probably already stopped writing raw API calls in a loop and moved to a harness — Claude Code, Codex, Pi, or something you wrote yourself. The harness owns the hard parts: tool dispatch, context management, retries, sandboxing, and increasingly, routing.

A good recent example of where this is heading is Omnigent, an open-source "meta-harness" from the Databricks AI team and Neon. It sits above the individual harnesses and gives you three things: composition (swap or combine Claude Code, Codex, Pi, and your own agents without rewriting), collaboration (shared live sessions), and — the part I care about here — control. In Omnigent, guardrails like spend caps, sandboxing, and model routing are enforced as stateful policies at the meta-harness layer, not baked into prompts.

I'm using Omnigent as the concrete example because it names the pattern cleanly, but the pattern is bigger than any one tool. LiteLLM, OpenRouter-style proxies, and most serious in-house agent stacks are all converging on the same idea: the routing decision lives in the harness, and the model is a swappable target behind it.

Once that's true, an obvious question follows.

## Not every step needs a frontier model

Here's the thing nobody building agents disputes once they look at their own traces: most agent steps are not hard.

A single agent task fans out into dozens of model calls. A realistic breakdown of that traffic looks something like:

- **Classification and routing** — "is this a bug report or a feature request?", "which tool should I call?"
- **Extraction** — pulling fields out of a document, parsing a log, structuring a response
- **Summarization** — compressing a file, a thread, or a tool output back into context
- **First-pass drafting** — a rough answer that a stronger model (or a human) will refine
- **Tool-call formatting** — turning intent into a well-formed function call

None of that needs a frontier model. An open-weight model in the 8B–70B range — Qwen3, Llama 3.3 70B, Mistral, a DeepSeek distill — handles it fine. The hard reasoning steps, the ones where quality genuinely separates the models, are a minority of calls even though they get all the attention.

So the routing policy writes itself:

> Route the bulk — the cheap, repetitive, high-volume steps — to a model you host yourself. Reserve frontier API calls for the genuinely hard steps that justify the price.

That's just "own the base, rent the spike" applied to inference. Except now the base is your baseline token volume and the spike is the frontier-model call for the step that actually needs it. And the mechanism that makes it real — the router — already exists in your harness.

I want to be precise about what I'm not claiming, because I'd rather you trust the parts that are true. I'm not telling you to route everything to Bit Refinery. You can't self-host Claude or GPT, and for the hard steps you shouldn't try — those calls go to the frontier APIs, and that's correct. The argument is narrower and, I think, stronger: for the routed-down tier, a flat-rate GPU you control beats per-token API pricing, and it beats it by a lot once your volume is real.

## The cost math, done honestly

This only matters if the numbers work, so let's run them instead of asserting them.

Bit Refinery hosts an NVIDIA RTX 6000 Ada (48 GB) at a flat $805/month — single-tenant, dedicated, no per-token metering, no egress fees. Flat is the whole point: your cost is fixed whether the GPU sits at 5% utilization or you peg it at 95% all month.

A single RTX 6000 Ada serving a quantized model in the 8B–14B class (the sweet spot for routed-down work) will comfortably push on the order of millions of tokens per day in batched, concurrent serving — the exact number depends on the model, quantization, sequence lengths, and batching, so measure yours rather than trusting a brochure figure, mine included. But the shape of the economics doesn't depend on hitting a precise throughput number. It depends on one thing: flat cost vs. per-token cost.

Here's the break-even logic. Per-token API pricing for small hosted models tends to land somewhere in the low tens of cents per million tokens on the cheap end, and higher for stronger models. Call the routed-down tier's blended rate P dollars per million tokens on a hosted API. Your Bit Refinery GPU is a fixed $805/month. The break-even volume is simply:

```
break_even_tokens_per_month = $805 / P  (in millions of tokens)
```

- If your routed-down tier runs at a blended API rate of $0.30 / million tokens, break-even is about 2.7 billion tokens/month (~90M/day).
- At $0.60 / million, break-even drops to about 1.3 billion tokens/month (~45M/day).
- At $1.00 / million, break-even is around 800 million tokens/month (~27M/day).

(Per-token prices move constantly, so treat these as a worked example of the method, not a live price sheet — plug in whatever your provider actually charges today.)

Below break-even, the API is cheaper and you should just use it — I'm not going to pretend otherwise. Above break-even, every additional token on the flat GPU is effectively free, while the API bill keeps climbing linearly with no ceiling. And here's the part that bites growing teams: agent workloads don't grow gently. The moment you go from a prototype to something running in a loop across real users, your routed-down token volume can 10x in a quarter. On per-token pricing that's a 10x bill. On a flat GPU, that's the same $805 — you just moved up the utilization curve you were already paying for.

That asymmetry — linear, uncapped cost on one side; fixed cost with a ceiling on the other — is the entire economic case. It's not "cheaper per token." It's "your worst-case cost is known, and it stops going up."


![Cost comparison chart showing linear API pricing vs flat-rate GPU hosting with a break-even point](/api/storage/files/blog-images/infographic-1783808174824.jpg)

## The other half: data you don't have to send anywhere

The economics get the top billing, but for a lot of teams the routing story is really a data story wearing a cost hat.

When the harness routes a step to a hosted frontier API, whatever's in that step's context — the internal document, the customer record, the proprietary code — leaves your environment and goes to a third party. For most steps, most teams are fine with that. But some steps carry data you'd rather never left: PII, PHI, regulated financial data, unreleased IP.

Routing gives you a clean way to handle that: send those steps to a self-hosted endpoint and they never touch a third-party model API at all. On Bit Refinery's single-tenant private hosting, the model runs on dedicated hardware in Denver or Seattle, there's no telemetry back to a model provider, and your data isn't training anyone's next release. The routing policy becomes a data-governance control: steps tagged sensitive route to the private endpoint; everything else routes normally.

For the Front Range startups I mostly talk to, this shows up earlier than you'd expect — usually the first time a design-partner enterprise sends a security questionnaire asking where their data goes. "Our sensitive steps run on single-tenant hardware we control, never on a shared API" is a much better answer than "we trust our API vendor's DPA."

## What the integration actually looks like

The reason this pattern is worth acting on now rather than filing away is that the plumbing is trivial. Self-hosted open models are served behind OpenAI-compatible endpoints, and every serious harness router speaks that protocol. So "route to Bit Refinery" is, mechanically, just pointing a base_url at your private endpoint for the routed-down tier.

Conceptually, a routing policy that sends the cheap, high-volume, and sensitive steps to your Bit Refinery endpoint and lets everything else hit a frontier API looks like this:

```yaml
# Illustrative routing policy — the pattern, not a verified schema.
# Omnigent is alpha and its exact config surface will change; check the
# current docs at omnigent.ai. The shape below is what harness routing
# looks like across most tools.

models:
  # Your self-hosted, flat-rate tier on Bit Refinery.
  # OpenAI-compatible endpoint on single-tenant hardware.
  - name: private-base
    provider: openai-compatible
    base_url: https://your-endpoint.bitrefinery.com/v1
    model: qwen3-32b
    api_key: ${BR_ENDPOINT_KEY}

  # Frontier tier for the genuinely hard steps.
  - name: frontier
    provider: anthropic
    model: claude-opus-4-8

routing:
  policies:
    # Anything you've tagged sensitive never leaves your hardware.
    - match: { tags: [sensitive] }
      route_to: private-base

    # High-volume, low-difficulty step types → self-hosted base tier.
    - match: { step_type: [classify, extract, summarize, format_tool_call] }
      route_to: private-base

    # Everything else → frontier.
    - default: frontier
```

The important line is `base_url`. That's the whole integration: your Bit Refinery Private LLM Hosting endpoint drops in exactly where a hosted model API would, and the harness routes to it by policy. No SDK swap, no rewrite — the harness was already going to make a routing decision; you're just adding a target it can point at.

I'll flag the honest caveat plainly: Omnigent is alpha and built in the open, so its exact policy YAML will move. Don't copy the block above expecting it to run — treat it as the shape of the pattern, confirm the current syntax against their docs, and if you want, we'll get on a call and wire your actual endpoint into whatever router you're running.

## Where this leaves you

If you're building agents on the Front Range and your API bill has started to sting — or your first enterprise customer just asked where their data goes — the move isn't to abandon frontier models. It's to stop sending them work they don't need to do.

1. **Look at your traces.** Count how many of your model calls are classification, extraction, summarization, and formatting. For most agent workloads it's the majority.
2. **Estimate that tier's monthly token volume.** Run it against the break-even math above at your provider's real per-token price.
3. **If you're above break-even — or heading there fast — route that tier to a flat-rate GPU you control.** Fixed cost, a ceiling on your worst case, and sensitive steps that never leave your environment.

That's "own the base, route the spike." The harness gives you the router. Open models give you the quality floor for the easy steps. A single-tenant RTX 6000 Ada at $805/month flat gives you the destination — and a cost curve that stops climbing right when your usage takes off.

If you want to pressure-test the numbers against your actual workload, that's the kind of conversation I like having. Get in touch and bring your traces.