--- title: "Haven — Private GPU Use Case | Bit Refinery" url: "https://bitrefinery.com/private-gpu-use-cases/haven" description: "Haven classifies personal data on-device and sends only the hardest 0.8% of cases to a Gemma 4 31B model on a dedicated Bit Refinery GPU. Measured at 4.0 s average over 654 requests." lastmod: "2026-08-24" source: "auto-generated from SSG HTML" --- [All use cases](https://bitrefinery.com/private-gpu-use-cases) LIVE IN PRODUCTIONConsumer security software # Haven Personal data security that never ships your messages to a vendor [havenprotect.app](https://havenprotect.app) At a glance Model Gemma 4 31B, FP8 Serving vLLM in a rootless container Hardware One 48 GB NVIDIA card Measured latency 4.0 s average Cold starts None — model stays resident The problem ## The scanner has to read the most private thing a person owns Haven is a desktop application that finds sensitive data a person has forgotten they exposed. A tax return attached to a chat thread four years ago. A password typed in plain text to a colleague. A photograph of a passport sent to a landlord. The premise only works if the software can look at message content, because that is where the exposure is. That creates an uncomfortable requirement. To tell a user that their own messages are safer with Haven installed, the product has to be able to say exactly where every message went — and “we send it to a large AI provider and they have promised not to keep it” is not an answer that survives the question. The architecture ## Three tiers, and only the third one leaves the device Haven’s answer is not to avoid inference. It is to make the off-device portion small, bounded, visible, and served from hardware the company controls end to end. Tier 1On the user’s machine Deterministic pattern matching. Card numbers, key formats, national ID shapes — anything a well-written rule can settle without a model. Runs on every message. Tier 2On the user’s machine A small bundled classifier, shipped as signed ONNX weights and updated over its own release channel. Also runs on every message. Between them, Tiers 1 and 2 resolve the overwhelming majority of traffic without a single byte leaving the device. Tier 3On a Bit Refinery GPU The tiebreaker. When the local classifier cannot reach a confident verdict — roughly 0.8% of messages in measured real inboxes — a bounded excerpt of at most 2 KB goes out in a single round trip, gets a verdict, and is discarded. Every one of those calls is visible to the user in the application’s own network activity log. Why it can’t be an API ## A retention policy is a promise. A private GPU is a property of the system. Every major inference provider will tell you they do not train on API traffic and do not retain it beyond an abuse-monitoring window. Those statements are almost certainly true. They are also unverifiable from the outside, revocable by a terms-of-service update, and subject to a subpoena served on someone other than you. Running an open-weights model on a dedicated card changes the category of the claim. There is no third party in the request path, so there is no third-party retention to promise about. The guarantee stops being a policy Haven asks users to trust and becomes a fact about where the bytes went. This is also why the carve-out is stated plainly rather than buried. Haven does not claim it never sees content — it claims that a bounded excerpt of a small minority of messages is evaluated on infrastructure it operates, and it shows the user each time that happens. > The data being classified is private message content, so a third-party inference API is disqualified on principle rather than on price. Owning the box is what makes the no-retention promise enforceable instead of contractual. On our side ## What the deployment actually is Gemma 4 31B in FP8, served by vLLM inside a rootless container, on a single 48 GB card. A reverse proxy in front handles admission control and TLS termination, and its request logging is content-stripped by configuration rather than by convention. The model is loaded once and stays loaded. 4.0 s Average request latency across 654 lifetime requests ~1.4 K Prompt tokens per Tier 3 call, by design — excerpts are bounded ~0.8% Of messages that reach Tier 3 at all The 48 GB budget turned out to have room in it. The server was originally configured to claim 92% of the card, most of that a pre-allocated key-value cache far larger than the real prompt sizes needed. Re-tuning that budget downward freed enough memory to bring a second, smaller model onto the same card without measurably moving the first one’s latency — which is a fair description of what a dedicated card buys you. On shared or serverless capacity that tuning decision is not yours to make. The cost argument ## At this volume, an API would be cheaper. That is not the point. Haven is in alpha. Tier 3 currently sees on the order of 53 requests a day. At roughly 1.4 K prompt tokens each, a commodity inference API would bill that at a rounding error, and it would be dishonest to present this deployment as a cost optimisation. It is not one yet. What the dedicated card buys today is the architectural guarantee, and it buys it at a price that does not move. The interesting property is what happens next: Tier 3 volume scales with installed seats, so a workload that is a rounding error at alpha becomes a real monthly line item at ten thousand users and a serious one at a hundred thousand. The card is already paid for at all three volumes. That is the trade — you pay early for capacity you have not filled, in exchange for a bill that stops being a function of growth. For a workload with a genuine floor, that crossover is the whole analysis. For a workload with privacy constraints, it is a bonus on top of a decision that was already made. What this case does not prove - Haven is a pre-release product. These are alpha-stage numbers from a small user base, not a capacity study. - Fifty-three requests a day does not exercise a 48 GB card. The latency figures say the deployment is healthy; they say nothing about its ceiling. - The privacy argument here is about infrastructure topology. It is not a security audit of the application, and it does not substitute for one. ## Have a workload shaped like this one? A resident open-weights model on a card you don't share is a different cost and privacy story than a per-token API. Tell us what you're running and we'll size it.