---
title: "Serving Layers vs. Agent Harnesses: How Many Agent Sessions Fit on One 48GB GPU"
url: "https://bitrefinery.com/blog/serving-layer-vs-agent-harness-48gb-gpu-concurrency"
description: "vLLM and llama.cpp on one RTX 6000 Ada, driven with agent-shaped load. Prefix caching cuts time-to-first-token 86x, the two serving layers decode within 1% of each other, and the real ceiling is four concurrent agent sessions - set by prefill compute, not VRAM."
author: "Bit Refinery Team"
date: "2026-07-27"
lastmod: "2026-07-27"
tags: ["gpu hosting", "llm", "ai infrastructure", "vllm", "benchmarks"]
source: "blog CMS"
---

# Serving Layers vs. Agent Harnesses: How Many Agent Sessions Fit on One 48GB GPU

Everyone asking "which open model should I self-host?" is usually asking the wrong question. The model is the easy part. What determines whether your setup works is the two layers wrapped around it — and most people conflate them.

So we put both on a card we own, ran them until they broke, and wrote down the numbers.

## The two layers, separated

**The serving layer** turns weights into an HTTP endpoint. vLLM, SGLang, Ollama, llama.cpp, TGI. Its job is to load the model, manage the KV cache, and speak something OpenAI-compatible. That's it.

**The harness** is the agent loop sitting on top. OpenHands, Aider, Cline, Goose, Continue, OpenCode. It decides what to put in the context window, calls tools, applies file edits, retries when things fail, and compacts history when it runs long.

The distinction matters because **the harness determines the load, and the serving layer determines whether that load survives.** A chat user sends 500 tokens. An agent working through a real repository sends 30,000 to 60,000 tokens *every turn*, because it re-sends the accumulated conversation each time. That difference is the whole ballgame, and almost nothing published about "running Gemma locally" accounts for it.

## Picking a harness by the job

Three jobs cover most of what people actually do:

- **Coding in a terminal** — Aider. It's scriptable, deterministic, and has the tightest context discipline of the bunch. Best choice if you want to know exactly what's being sent.
- **Coding in an editor** — Cline or Continue. Lower friction, but they index aggressively, which means larger and less predictable contexts.
- **General tool use and documents** — Goose. Broader tool surface, less code-specific.

We're not going to rank all eight. That post exists everywhere and it's stale in four months. What doesn't exist is the number underneath all of them: **how many of these can one card actually run at once?**

## The test

A single NVIDIA RTX 6000 Ada, 48 GB (49,140 MiB), driver 595.71.05, CUDA 13.2, Ubuntu 24.04, 8 vCPU, 62 GB RAM. **A genuinely clean card** — 2 MiB in use before we started. (Our earlier [LM Studio Bionic benchmark](/blog/lm-studio-bionic-remote-gpu-benchmark) ran with a 4B vision classifier holding 9.8 GB the whole time, and we flagged those as shared-card numbers. This run settles that debt.)

One model, quantized two comparable ways so the serving layers face the same work:

| Serving layer | Build | Size |
|---|---|---|
| llama.cpp (`server-cuda` container) | Qwen3-Coder-30B-A3B-Instruct **Q4_K_M GGUF** | 18.63 GB |
| vLLM 0.25.1 | Qwen3-Coder-30B-A3B-Instruct **4-bit AWQ** (group 32) | 17 GB |

Both at 131,072 context. Both 4-bit. The GGUF is the same artifact LM Studio served in our earlier test, so the llama.cpp leg is directly comparable to it.

Prompts were **agent-shaped, not chat-shaped**: a large tool-definition system prompt, four whole source files pasted in as gathered context, then a short instruction — assembled to hit a target token count. Chat-shaped prompts produce flattering numbers that don't survive contact with a real harness.

Then we checked that assumption against an actual harness. We pointed **Aider 0.86.2** at the same endpoint with a real Go module — 8 files, 5,512 lines, 100 KB — and asked it to find a goroutine leak. It found the missing `close()` and the cancellation cleanup correctly. What it sent:

| Aider context | Tokens sent |
|---|---|
| 3 files | **12k** |
| 8 files (the whole module) | **30k** |

That's turn one. Every turn after adds the prior exchange. So a working agent session lands in the 30–60K band, and the 45K we used for the load tests is squarely representative — not a number we picked to make the hardware look busy.

## Finding 1: prefix caching is worth more than your GPU choice

The first run produced a 200× spread between median and p95 time-to-first-token. That wasn't variance — it was our own bug. We sent identical prompts across rounds, so round one paid full prefill and the rest hit the prefix cache.

Fixing it turned out to be the most useful thing we measured, because **both numbers are real and they describe different moments in an agent session.** Turn one is cold. Every turn after that reuses the same prefix, so it's warm.

llama.cpp, single session:

| Context | TTFT **cold** (turn 1) | TTFT **warm** (turn 2+) | Decode tok/s |
|---|---|---|---|
| 3,048 | 1.00 s | 0.05 s | 193.8 |
| 15,215 | 3.85 s | 0.08 s | 153.4 |
| 45,687 | **17.68 s** | **0.21 s** | 98.6 |
| 101,530 | **74.13 s** | **0.38 s** | 60.8 |

At a realistic 45K agent context, prefix caching is the difference between **17.68 seconds and 0.21 seconds** to first token. That's 86×. At 100K it's 195×.

Decode speed is identical in both columns, as it must be — caching only affects prefill.

The practical consequence: a harness that appends to a stable prefix lives in the right-hand column. A harness that rewrites, reorders, or aggressively compacts its context invalidates the cache and pays the left-hand column *every single turn*. When you're choosing a harness, that behavior matters more than its feature list.

## Finding 2: vLLM and llama.cpp decode at exactly the same speed

This surprised us.

| Context | vLLM TTFT (cold) | llama.cpp TTFT (cold) | vLLM decode | llama.cpp decode |
|---|---|---|---|---|
| 3K | 0.22 s | 1.00 s | 189.1 | 193.8 |
| 15K | 1.51 s | 3.85 s | 151.9 | 153.4 |
| 45K | **8.61 s** | **17.68 s** | 99.6 | 98.6 |
| 100K | **37.3 s** | **74.1 s** | 60.9 | 60.8 |

Decode throughput is within 1% at every depth. If you are one person with one session, **the serving layer barely matters for token generation speed.**

vLLM's advantage is entirely in prefill — consistently about **2× faster** at every context depth. For an agent that's re-reading a large context on every cold turn, that's the half that hurts.

## Finding 3: the ceiling is compute, not memory — and it's 4 sessions

Here's the number nobody publishes. We ran N concurrent agent sessions at 45K context each, warm, and measured aggregate throughput:

| Concurrent sessions | Wall/round | TTFT median | Aggregate tok/s | Errors |
|---|---|---|---|---|
| 1 | 7.0 s | 4.41 s* | 36.7 | 0 |
| 2 | 9.1 s | 0.33 s | **56.3** | 0 |
| 4 | 17.9 s | 0.50 s | **57.3** | 0 |
| 6 | 37.1 s | 10.75 s | 41.4 | 0 |
| 8 | 82.1 s | 33.3 s | 24.9 | 0 |
| 12 | 153.6 s | 77.0 s | 20.0 | 0 |

\* blends one cold round with one warm round.

![Aggregate throughput against concurrent agent sessions on one RTX 6000 Ada: vLLM peaks at 57 tokens/sec at 4 sessions then collapses, while llama.cpp stays flat near 8.5 and fails up to 75% of requests](/api/storage/files/blog-images/blog-1785121136271-serving-concurrency.png)

**Aggregate throughput peaks at 2–4 concurrent sessions and then collapses.** At 4 sessions, median time-to-first-token is still half a second. At 6, it jumps twenty-fold to 10.75 seconds. The knee is sharp.

We expected to find a memory wall. We didn't. vLLM sized its KV pool to **259,584 tokens** and reported a maximum concurrency of 1.98× for full 131K requests — and that pool was *never exhausted*. Zero OOMs at any level. Peak VRAM was 44.3 GB of 48.

What actually runs out is **prefill compute**. Under the cold sweep, wall-clock time scaled almost perfectly linearly with session count (11.7 → 23.0 → 47.0 → 74.6 → 100.9 → 154.6 seconds) while aggregate throughput stayed pinned near 21 tok/s. Adding sessions bought nothing; it just queued everyone.

So the sizing rule for agent workloads is not "how much VRAM do the weights need." It's **how many prefills per second do you need**. Budget roughly **4 concurrent agent sessions per RTX 6000 Ada** at 45K context.

Under that load the card drew **306.98 W against a 300 W cap at 77 °C** — versus 219.8 W and 69 °C in our single-session Bionic run. Stacking agents is what finally makes this card work.

## Finding 4: llama.cpp doesn't slow down under load, it returns 500s

We assumed llama.cpp would degrade gracefully and vLLM would be the brittle one. **We had it exactly backwards.**

```
cold N=4  →  4 ok, 4 failed   HTTP 500: "Context size has been exceeded."
cold N=6  →  8 ok, 4 failed
cold N=8  →  8 ok, 8 failed
cold N=12 → 12 ok, 12 failed
```

llama.cpp's `-c 131072` is a **unified pool shared across all four slots**, not per-slot. Four sessions at 45K need 180K tokens. The pool holds 131K. Half the requests die outright.

Side by side at 4 concurrent sessions — the realistic "small team" case:

| | vLLM | llama.cpp |
|---|---|---|
| Requests succeeded | **8 / 8** | **2 / 8** |
| Wall per round | 17.9 s | 121.0 s |
| Aggregate tok/s | **57.3** | 8.5 |

llama.cpp's aggregate throughput sits at ~8.5 tok/s regardless of how many sessions you add, with 50–75% of requests failing. vLLM ran the same load to twelve sessions without a single error.

This is fixable — raise `-c`, or set `--parallel` deliberately. But **you have to know to do it.** vLLM sizes its KV pool to available VRAM automatically; llama.cpp allocates exactly what you asked for and hard-fails when that isn't enough. If you put llama.cpp in front of a team without tuning it, you will not see it get slow. You will see 500s.

## Finding 5: operational cost is real and it favors llama.cpp

For a single developer, the setup difference is the opposite of the performance difference.

llama.cpp was serving requests **23 seconds** after `podman run`, with no configuration. vLLM cost us two failed starts before it came up at all: a renamed CLI flag (`--disable-log-requests` became `--no-enable-log-requests` in 0.25.1) and a missing `ninja` binary that only surfaces when it JIT-compiles kernels for this MoE architecture. Once up, its own init took 33.4 seconds.

Worth knowing: **there is no prebuilt Linux CUDA binary for llama.cpp.** Upstream ships CUDA builds for Windows only. Use the official `ghcr.io/ggml-org/llama.cpp:server-cuda` container rather than installing a 4 GB CUDA toolkit to build it yourself.

## What this means for sizing

- **One developer, one session.** Either layer. Decode speed is identical, llama.cpp is far easier to stand up. Plan around **~99 tok/s at 45K context**, and make sure your harness keeps a stable prefix so you live at 0.21 s TTFT instead of 17.68 s.
- **A team sharing one card.** vLLM, and budget **4 concurrent sessions per RTX 6000 Ada**. Past that, throughput falls and latency becomes unusable — you need a second card, not a bigger one.
- **Size for prefill, not VRAM.** A 30B model at 4-bit leaves plenty of memory headroom on a 48 GB card. Memory is not what you run out of.

## Where you still route out

None of this argues for self-hosting everything. A 30B model at 4-bit is a capable coding assistant, not a frontier model, and there are turns it shouldn't handle. The economics of running a private endpoint as your baseline tier and routing the hard turns to a frontier API is a separate argument, and we made it in [Own the Base, Route the Spike](/blog/own-the-base-route-the-spike-where-private-gpus-fit-in-a-model-routing-world).

This post is the other half: you already decided to self-host, and you need to know what one card actually holds.

For that workload, Bit Refinery's Private GPU Cloud runs single-tenant NVIDIA GPU VMs with full root and SSH access — RTX 6000 Ada 48 GB from **$790/month**, RTX PRO 5000 Blackwell 48 GB from **$840/month**, up to 7 GPUs per pod with NVLink. Flat monthly billing, no per-second metering, $0 egress with 10 TB included, Tier 3 data centers in Denver and Seattle.

Every number above came off a card in that fleet. If you want to check our arithmetic, [spec a pod](/services/private-gpu-cloud) and run the same sweep yourself.