vLLM versus llama.cpp on a GB10

I have been running llama.cpp on my GB10, a DGX Spark with 121GB of unified memory, for weeks. The engine I was defending was not as fast as its own logs.

Round 3 of my private engine tournament pitted the latest vLLM against that production llama.cpp build. Same model both times, Qwen3.8-Flash-Next, a large mixture of experts with about 6B active parameters, MTP speculative decoding on both sides, temperature zero. vLLM won. But the win is smaller than the discovery underneath it: the baseline number I had been quoting for weeks was a measurement artifact, and the race had been scored wrong the whole time.

The rounds have a history: Round 1, this site's earlier posts, was vLLM finding its footing on the Spark. Round 2 was the switch to llama.cpp, which served the box for weeks. Round 3 is the rematch with the current vLLM, and this time I brought a scoreboard.

Round 3: vLLM versus llama.cpp on a GB10

How the test ran

One benchmark would have lied to me, so I ran a battery, and then re-ran the parts that disagreed with themselves.

Every number below came through the chat endpoint, the path my automations actually use, measured as client wall time: timer starts before the request leaves, stops when the last token is accounted for. Temperature zero, medians of three to five runs. On the vLLM side, the official release-branch image with the vendor's hybrid NVFP4 checkpoint, worst per-tensor relative error 0.035, under the 0.05 acceptance line. On the llama.cpp side, the exact production unit, same quantization, same context settings. I stopped the production unit for the measurement window so neither engine shared the box mid-run.

The first pass showed a two percent win for vLLM, inside noise. I was about to write that off when the two engines' timing reports disagreed about basic arithmetic.

The finding that mattered more than the win

llama.cpp's timing report excludes the draft verification steps of its own speculative decoding. When the drafter is getting rejected a lot, that number is not conservative. It is fictional. On one identical request, the engine reported 49.48 milliseconds per token, 20.2 per second. Same request, client wall time: 621 tokens in 30.09 seconds, 20.6 per second. Close enough. But the historical 23.8 median I had been treating as the number to beat was the biased metric at its flattering best.

vLLM's usage-based wall time is honest. Once both engines were measured by what the client feels, the tie evaporated and the race got re-scored.

The numbers

Chat endpoint, client wall time, temperature zero.

| Metric | llama.cpp | vLLM (NVFP4 hybrid, MTP=2) | |---|---|---| | Single stream, warm | 20.6 to 21.4 tok/s | 24.3 tok/s | | Four concurrent requests, aggregate | 20.0 tok/s | 43.6 tok/s | | Concurrency gain vs its own single stream | 1.0x | 1.8x | | Cached prefill | cache keep rate 6.8%, thrash | 5,850 tok/s on a prefix hit | | RAM footprint | 40 GiB | ~100 GiB | | Cold start | ~100 s | ~6 to 10 min | | Spec-decode acceptance (chat) | 0.37, mean len 2.48 | 0.54 to 0.58, len ~2.1 |

Single-stream, vLLM is up about 15 to 20 percent over a baseline that was already corrected upward. The concurrency number surprised me more than it should have. llama.cpp was never a concurrent server; I chose it precisely because my workload is one agent answering at a time, and I had never bothered to test what happens when three show up at once. I tested it this time. The four streams finished at 20.3, 9.7, 6.5 and 5.0 tokens per second, mostly one after another, aggregate throughput no better than a single stream. vLLM served all four at once, roughly 11 per stream each, 43.6 per second in total. The batching story is real. The honest framing is that I have been running a single-lane road and only now noticed the traffic.

The iterations

The MTP draft depth sweep was cheap to redo, so I did it three times on the vLLM side. Depth one measured worse than depth two on both endpoints, 22.4 versus 24.3 on chat, even though per-token draft acceptance was higher, 72 to 74 percent against 54 to 58. With depth one, acceptance is high but short, 1.7 tokens per cycle. With depth two, acceptance is lower but each hit pays for two tokens, 2.1 per cycle. Optimize accepted tokens per cycle, never acceptance rate. Depth three looked flat on the chat path, so the community's "stop at two" cap is doing exactly what it says on this box too.

On the llama.cpp side, the loss mechanism was equally mechanical. The prompt cache on the configured context is only 358 megabytes, and between four test prompts the log just gives up: making room for a prompt cache entry, removing the oldest entry. The slot selector kept 6.8 percent of a re-sent prompt. Meanwhile the four concurrent requests all queued on the same slot. Each problem, individually boring. Together, a loss.

vLLM's 471k-token KV pool absorbed the whole workload, and on a warm prefix cache it re-prefilled 10.7k tokens in 1.83 seconds, about 5,850 tokens per second, against 77 tokens per second cold. My automations re-send big prompts all day. Caching is load-bearing infrastructure, not a feature.

Determinism came out identical across both engines: identical first-token logprobs run to run, which for my use is a requirement, not a bonus.

The honest costs

A win you cannot price is not a win.

RAM: about 100 GiB against 40. On a 121GB unified-memory box that is a real tax, and co-residency with a browser and a desktop is tight. If the OOM risk gets annoying, the memory fraction comes down and context comes down with it.

Restarts: ten minutes cold against under two. The rollback path is keeping the old llama.cpp unit installed, disabled, so undoing this is a two-minute job.

Per token per gigabyte, llama.cpp still wins, roughly 2.5 to 1. If your box runs one thing and nothing else, this race goes the other way. My box runs everything.

One line I have not earned yet: an alternative third-party quant ladder with an advertised 50 to 60 tokens per second, untested, provenance I cannot vouch for. And a 16k long-context test that came back invalid for both engines, because reasoning tokens were counted as decode. Both stay off the scorecard until they are measured honestly.

Production switched to vLLM, conditionally, with the old engine kept one command away. Round 3 complete.