Benchmarks - Private Local AI
One number per cell, from 1 to 99: how much of what an engine can do this machine is actually able to use. It is our index, not a measurement - the figures behind it are published benchmark numbers we have read and runs we made ourselves on 32 GB, 64 GB, 96 GB boxes, a single RTX 3060, two RTX 5070 Ti and a mixed-GPU machine. Reading it is quick; arguing with it in detail is the appendix.
80-99 - the engine's full working range opens up on this box
50-79 - it runs, with configuration work and a real drop in what you can serve
20-49 - only smaller or quantised models reach usable speed here
under 20 - it does not belong on this hardware today
The index is built from three scored parts, weighted as they matter to a client asking whether their box will do the job: whether the weights can be loaded at all (40%), how close the machine gets to that engine's expected speed - taken from published runs and ours (35%), and how much setup the engine asks for (25%). A cell carries a tag so you know what it rests on: own is our run, pub is a figure somebody else published and we can name the source for, ext is our extrapolation of a configuration close to one we have numbers for. Scores are not comparable across rows in any strict sense: two engines on one box are only comparable when the model file and prompt length were held identical, which is what the appendix rows show.
| Hardware | llama.cpp | vLLM | Sonar | ExLlamaV3 | Strata | PolyStrata | Avalon | Xolotl |
|---|---|---|---|---|---|---|---|---|
| 32-1-4070lap | 58pub | 9ext | 6ext | 12ext | 74pub | 86pub | 31ext | 44pub |
| 32-1-3060 | 41pub | 8ext | 6ext | 24ext | 88pub | 79ext | 46pub | 72pub |
| 64-1-3060 | 52pub | 10ext | 7ext | 26ext | 94pub | 84pub | 58pub | 81pub |
| 64-1-5070 | 49pub | 14ext | 9ext | 28ext | 90pub | 88pub | 55pub | 63pub |
| 96-2-5070ti | 66pub | 21pub | 17ext | 35ext | 92pub | 94pub | 71pub | 90pub |
| 96-2-mixed | 61pub | 12ext | 8ext | 19ext | 86pub | 93pub | 88pub | 57ext |
| 110-1-4090 | 57pub | 26pub | 19ext | 44ext | 89pub | 91pub | 70pub | 96pub |
| 128-1-8060s | 47pub | 7ext | 6ext | 11ext | 83pub | 76ext | 52ext | 60ext |
| 128-1-m3u | 54pub | 11ext | 9ext | 23ext | 37ext | 33ext | 12ext | 26ext |
| 128-1-h100 | 63pub | 92pub | 88pub | 30ext | 41ext | 38ext | 29ext | 35ext |
What the codes mean
Column names are the repository names: Sonar is what Aphrodite Engine calls itself now, and ExLlamaV3 is the current ExLlama - its cells are all marked ext because the only published runs on these boxes are from the archived V2. Ollama and exo are not columns here: one is a wrapper over llama.cpp, the other only moves a model between separate machines. Both have articles on the blog, and their published figures are kept in the appendix below, which is where the sources live.
[system DDR]-[number of GPUs]-[GPU model]: 64-2-3060 is 64 GB of system RAM with two RTX
3060 cards. A box that is not a plain desktop tower - a laptop, or an Apple/Strix Halo machine where one pool of memory serves both
CPU and graphics - is marked in the map, because the notation describes DDR and discrete cards only.
32-1-4070lap- laptop - 8 GB GPU, 32 GB RAM - RTX 4070 Laptop, i7-14700HX, 32 GB, Windows. Our smallest test box.
32-1-3060- desktop - 1× RTX 3060 12 GB, 32 GB RAM - Core Ultra 7 265, 46 GB in the box we ran; 32 GB is the floor where this model still loads.
64-1-3060- desktop - 1× RTX 3060 12 GB, 64 GB RAM - the same card with more headroom: weights move into system RAM instead of NVMe.
64-1-5070- desktop - 1× RTX 5070 12 GB, 64 GB RAM - a current mid-range card, 64 GB system RAM.
96-2-5070ti- two cards - 2× RTX 5070 Ti 16 GB, 96-128 GB RAM - the shape most people actually build: two consumer cards and a lot of memory.
96-2-mixed- mixed GPUs - RTX 7900 XTX 24 GB + R9700 32 GB, 96 GB RAM - an AMD pair and an Intel CPU of an older generation: the case where driver behaviour matters.
110-1-4090- single big card - RTX 4090 24 GB, 110 GB RAM - one high-end card with a full set of DIMs: the configuration our own agent stack was built on.
128-1-8060s- no discrete GPU - Ryzen AI Max+ 395, 128 GB unified - a Strix Halo box: one memory pool shared by CPU and GPU.
128-1-m3u- Mac mini / Mac Studio - Apple silicon, 64-512 GB unified - the only consumer box where an engine grows past one machine cleanly.
128-1-h100- server card - H100 80 GB / A100-class, 128 GB RAM - shown only to make the opposite point: what a data-centre card is for.
The published figures behind this index are listed, with hardware and version, in the appendix below. We keep them there rather than up front because they are not what most readers ask first.
The published figures behind the index, and where they came from
| Hardware (as the source describes it) | llama.cpp | Ollama | vLLM | Sonar (formerly Aphrodite Engine) | exo | Strata | PolyStrata | Avalon | Xolotl |
|---|---|---|---|---|---|---|---|---|---|
| RTX 4090 24 GB, rented (vast.ai), Ubuntu 24.04, CUDA 13.2, single user then 32 concurrent requestsQwen3.8-27B. llama.cpp and Ollama: unsloth GGUF UD-Q4_K_M (16.5 GB). vLLM: RedHatAI INT4 W4A16 GPTQ (19.5 GB). 4,096-token context | single-stream 49.5 tok/s; 32 concurrent 306 tok/s; first token 6.5 s2 | draft off: 39.9 tok/s single-stream, 39 at 32 concurrent, first token 101.7 s. With its default MTP draft: 76.22 | 54.3 tok/s single-stream; 300 at 32 concurrent; first token 8.6 s2 | — | — | — | — | — | — |
| L40S 48 GB, rented, same harness and settings as the row aboveQwen3.8-27B, same weights and 4,096-token context as above | 39.1 tok/s single-stream; 241 at 32 concurrent; first token 6.4 s2 | draft off: 20.0 / 20; first token 200.3 s. With MTP draft: 57.32 | 43.7 tok/s single-stream; 554 at 32 concurrent; first token 3.0 s2 | — | — | — | — | — | — |
| RTX 5090 32 GB, rented, same harness and settings as aboveQwen3.8-27B, same weights and 4,096-token context as above | 77.8 tok/s single-stream; 369 at 32 concurrent; first token 6.6 s2 | draft off: 76.8 / 74; first token 54.1 s. With MTP draft: 129.72 | 85.7 tok/s single-stream; 936 at 32 concurrent; first token 2.3 s2 | — | — | — | — | — | — |
| RTX 5090 32 GB, rented cloud instance, Ubuntu 24.04, driver 570.195.03, CUDA 12.8Llama 3.1 8B Instruct, Q4_K_M GGUF - the same file for both engines, 4,096-token context | 249 tok/s generation; 13,561 tok/s prompt; 83 ms to first token3 | 108 tok/s generation; 13,426 tok/s prompt; 221 ms to first token3 | — | — | — | — | — | — | — |
| Desktop: i7-8700, 64 GB DDR4-2400, RTX 5060 Ti 16 GB wired at PCIe 3.0 x8, Windows Server 2025Qwen2.5-Coder-7B Q4, identical model across runners; twelve prompts, 256 tokens out each | 77.0 tok/s (run directly)4 | 69.1 tok/s (the same weights through the Ollama wrapper)4 | — | — | — | — | — | — | — |
| Apple M3 Max, 30-core GPU, 36 GB unified memoryQwen2.5-Coder-7B Q4, same test suite as the desktop row above | 53.5 tok/s4 | 46.2 tok/s4 | — | — | — | — | — | — | — |
| 4 × M3 Ultra Mac Studio, 1.5 TB unified memory total, Thunderbolt 5 with RDMA enabledQwen3-235B at 8-bit and Kimi K2 Thinking, run across all four machines as one cluster | the article ran llama.cpp's RPC layer-spreading on the same machines and reported it got slower as nodes were added, without publishing a number for it6 | — | — | — | about 32 tok/s on Qwen3-235B and around 30 on Kimi K2 Thinking, measured by the journalist who wrote the piece; he also notes RDMA cut memory-access latency from about 300 µs to under 50 µs6 | — | — | — | — |
| 4 × NVIDIA RTX PRO 6000 Blackwell Server Edition, connected by PCIe only (no NVLink), CPU pinned with taskset 0-31DeepSeek-V4.1-Flash-1017, tensor parallel across four cards, FP8 KV cache, DSpark speculative decoding with 7 draft tokens; prompts cached, 2,048 tokens out | — | — | — | aggregate decode: 167.2 tok/s before tuning → 432.0 after at C=1, and 428.3 → 1,083.3 at C=8, for a 2,674-token prompt. Their headline claims 2.53-3.83× the baseline; self-reported7 | — | — | — | — | — |
| H100 SXM5 80 GB, single card, rented bare metal, driver 590.48.01, CUDA 13.0Llama-3.3-70B-Instruct at FP8 (--quantization fp8), vLLM v0.18.0; 200 prompts averaging 512 in / 256 out, run for three minutes after a warm-up | — | — | measured by the hosting company: 120 tok/s at one request, 650 at ten, 1,850 at fifty, 2,400 at one hundred; first token 380 ms at fifty requests; 71 GB idle, 78 GB peak8 | the same publisher quotes 48 / 380 / 1,450 tok/s for concurrency 1 / 10 / 50 - but the article says those figures are scaled estimates, not measurements of Aphrodite9 | — | — | — | — | — |
| GB200 and GB300/B300 racks (data-centre hardware, shown for completeness - not a workstation or gaming box)DeepSeek R1 / V3.2 at NVFP4, prefill and decode disaggregated; vLLM's own benchmark runs | — | — | 26.2K prefill TPGS and 10.1K decode TPGS on eight GB200; 7,360 tokens/GPU/s single-GPU prefill on a GB300/B300 and 2,816 output tokens/GPU/s in a mixed 2k-in/1k-out run. TPGS is their own metric: tokens per GPU per second, prefill included10 | — | — | — | — | — | — |
| RTX 5070 12 GB, 64 GB RAM desktopQwen3.8-Flash-Next at IQ2_XS, 32K prompt | — | — | — | — | — | the measurement PolyStrata cites for its own Qwen3.8 path: 79 tok/s writing, a 32K prompt read at 2,090 tok/s11 | 94-100 tok/s writing and 1,690 reading is the row for its own model on an 8 GB laptop card, not this box - see the laptop row below11 | — | community run on a 5070 with 64 GB: prefill 121-231 tok/s at 4-8K, decode 4.7 tok/s for DeepSeek-V4.1-Flash15 |
| RTX 3060 12 GB, Core Ultra 7 265, 46 GB RAM, Linux - a small card and not enough RAM for the weights eitherQwen3.8-Flash-Next GSQ-RCO IQ3_XXS (70.6 GiB), 36,864-token context, prompts of 4,000 and 32,000 tokens, 256 tokens out | decode 9.1-9.8 tok/s at 4K and 8.7-8.8 at 32K with tuned -ngl 99 -ncmoe; prompt 294-338 tok/s; with --fit on: 162 prompt / 9.6 decode1 | — | — | — | — | decode 41.6-41.9 tok/s at 4K and 41.8-38.8 at 32K; prompt 910-1,050 tok/s - the same run in the same file that produced the llama.cpp numbers on the left1 | — | — | community run at Q2_0 on a 3060 with 64 GB: prefill 882-981 tok/s across a 4K-128K sweep, decode 39-45 tok/s15 |
| RTX 4070 Laptop 8 GB, Core i7-14700HX, 32 GB RAM, Windows 10 - the smallest card in our set of testsQwen3.6-35B-A3B UD-Q3_K_M (17.1 GB), same file for both engines, 36,864-token context, 4K and 32K prompts | tuned with two MTP draft tokens: 44.4 tok/s writing at 4K, 41.5 at 32K; prompt 483-494. Without MTP: 44.4 / 39.2 and prompt 502-51712 | — | — | — | — | — | 94-100 tok/s writing and 1,690 reading, from its README table for this laptop. A community run on an RTX 4070 Ti SUPER with 30.9 GiB reported 165 tok/s short-chat and 3,287-3,615 reading11 | — | community run on a 4070 with 32 GB (Strata engine, IQ2_XS): 48 tok/s accepted decode15 |
| RX 7900 XT 20 GB, Ryzen 7 3800XT, 64 GB DDR4-3200, PCIe 3.0 - the case where system RAM matters more than VRAMQwen3.8-Flash-Next IQ3_KS hybrid (~89 GiB), experts in system RAM, 32K and 256K context, 512 tokens out | — | — | — | — | — | 51.1 tok/s short prompt without MTP, 68.4 with; 48.3 at a filled 250K context; prompt up to 1,453 tok/s - the comparison published in Avalon's README13 | — | 66.1 without MTP and 98.4 with; 57.3 at a filled 250K; prompt up to 1,854 tok/s. Its README also shows what happens when RAM is cut: 48.8-50.3 tok/s with 16 GB effective RAM, against 99.1 with the full 64 GB13 | — |
| rented RTX 4090 24 GB, AMD EPYC 7443 (24 cores), PCIe 4.0 x16Qwen3.8-Flash-Next, same weights and prompts as the row above | — | — | — | — | — | 141.4 tok/s with MTP, 101.5 without; after a 19K prompt 130.8 with prefill 3,422; at a filled 250K 128.5 with prefill 4,31913 | — | 141.9 with MTP, 98.0 without; after a 19K prompt 137.7 with prefill 3,113; at a filled 250K 123.8 with prefill 2,98613 | — |
| RTX 4090 24 GB, Ryzen 7 5700X, 110 GB DDR4 (90 GB until 1 Nov), one NVMe, Linux - the closest match to a two-card gaming box with lots of RAMDeepSeek-V4.1-Flash-1017 at its V4 XQ weights (552B parameters total, experts offloaded to RAM and NVMe) | a community run on a 5090 D v2 with 128 GB at UD-Q4_XS (136.7 GB) reported 8.6-10.6 tok/s; see the two-card row below for the tuned llama.cpp numbers17 | — | — | — | — | — | — | — | 32.6 tok/s decode measured inside agent tasks; 41.1 in its fast mode; prefill 258-439 tok/s - its README's Gauntlet v3 table, run on the machine described in this row14 |
| 2 × RTX 4090 24 GB, same CPU as above - the 'two combined gaming GPUs' caseQwen3.8-Flash-Next Q4_0 (93.7 GiB total, 65.4 GiB of experts) | generation 34.78 tok/s plain → 64.39 with -cmoe --moe-cache-mib 15000; prompt processing fell from 440.7 to 321.5 in the same test16 | — | — | — | — | community run on 2 × TITAN RTX 24 GB (125 GiB RAM) at IQ3_S: decode 70.9-77.3 tok/s, prompt 868.9-1,626.9 across 4K-128K18 | a run on 4 × RTX 5080 (125 GiB DDR5) at GLM 5.3 Flash 3.0-bit reported 64 tok/s short-chat, 51-72 range, and prompt 479-573 - a four-card box, not two17 | its README reports an RTX A4000 ×2 pipeline test at Qwen3.8: prefill of a 19K prompt 1,190 plain → 1,669-1,680 with its pipeline stage13 | speed-log rows on one 5090 with an EPYC 9654 (not the two-card layout): pooled decode 40.5-42.0 → 48.2-53.3 tok/s (+18.98% / +27.10%)14 |
| RTX 5090 32 GB, Xeon Platinum 8470Q (25 vCPU), 90-120 GB RAM - a workstation, not a gaming PCas published in PolyStrata's README table for this box: LFM2.5-24B-A2B up to DeepSeek V4 Flash UD-IQ2_XXS | on a 5090 with an EPYC 9654 and uniform Q3_K experts, Xolotl's log records 40.5-53.3 tok/s pooled decode of the same model family - see the row below14 | — | a single-user 8B or 70B run on a consumer card is not comparable to this box; vLLM's own GB200/GB300 rows are above10 | — | — | community run on a Strix Halo (Ryzen AI Max+ 395, 128 GB unified): 19-21 tok/s code, 11 chat; prompt 87-125 - no discrete GPU at all19 | 480-485 tok/s writing and 23,390-23,460 reading for LFM2.5-24B-A2B; 56 / 1,040 for DeepSeek V4 Flash at UD-IQ2_XXS; 59-66 / 820 for GLM 5.3 Flash; 189-192 / 4,980 for Kolibri Q2_K11 | community run reported in its README on an RX 7900 XTX + Ryzen 9 5900X (64 GB DDR4): median 116.5 tok/s generation, prefill 2,024-2,04313 | the run cited in the first cell of this row is its own: 40.5 → 53.3 tok/s pooled decode after tuning its expert pool on a 509014 |
| RX 7900 XTX 24 GB + Radeon AI PRO R9700 32 GB, layers split across the two cards, i5-14600KF, 96 GB DDR4-3200, LinuxOrnith 1.5 122B Q5_K_M (82.6 GiB), same file, same session, 128K context; first prompt of 98K tokens then follow-up turns | the first 98K-token prompt took 527 s to read; later turns 6.9 s; writing 13.5 tok/s at 100K context (Vulkan)17 | — | — | — | — | community run on a single RX 9070 16 GB (Ryzen 7 5800X3D, 64 GB) at IQ3_XXS: decode 57.9-58.7 tok/s, prompt 586-833 - a different box, shown so the trend is visible18 | the same prompt on the same two cards read in 184 s instead of 527 s (a 2.9× difference in prompt processing)17 | its README reports a single R9700 with an older Xeon at 262K context: 86.6 tok/s (range 85-93) with MTP, prefill 1,522-1,906; Strata on the same machine 31.013 | — |
Sources
- 1 https://github.com/Niko1221/Strata/blob/main/bench/results/2026-10-06-llamacpp-0.6.0/README.md (Strata team's head-to-head of Strata against llama.cpp v0.6.0, 6 Oct 2026)
- 2 https://computingforgeeks.com/ollama-vs-vllm-vs-llama-cpp/ (three-way test on rented cards, measured Oct 2026, published 6 Oct 2026)
- 3 https://atomic.chat/blog/guides/ollama-vs-llamacpp (Ollama v0.33.2 vs llama.cpp on an RTX 5090, 26 Jun 2026, updated 2 Sep 2026)
- 4 https://inventivehq.com/blog/ollama-vs-llama-cpp-vs-lm-studio-benchmark (Ollama vs llama.cpp vs LM Studio on an old desktop and an M3 Max, 26 Jun 2026)
- 5 https://raw.githubusercontent.com/turboderp-org/exllamav2/master/README.md (ExLlamaV2's own table comparing it with ExLlama V1)
- 6 https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5 (four M3 Ultra Mac Studios running exo, 18 Dec 2025)
- 7 https://x.com/dphnAI/article/2118450169154851066 (dphnAI's own Sonar measurement on four RTX PRO 6000 Blackwell, 5 Nov 2026)
- 8 https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/ (vLLM measured on a rented H100; undated page citing Nov 2026 pricing)
- 9 https://www.spheron.network/blog/deploy-aphrodite-engine-gpu-cloud/ (Aphrodite/Sonar figures that the article itself calls modelled estimates, not measurements)
- 10 https://blog.vllm.ai/2026-02-03/dsr1-gb200-part1.html and https://blog.vllm.ai/2026-02-13/gb300-deepseek.html (vLLM's own GB200 and GB300 numbers)
- 11 https://github.com/simplexai-labs/PolyStrata (README benchmark table for the laptop and the 5090 workstation, main branch as of 11 Nov 2026)
- 12 https://github.com/simplexai-labs/PolyStrata/blob/main/bench/results/2026-10-11-laptop-qwen36-llamacpp-b11540/README.md (PolyStrata team's head-to-head against llama.cpp b11540 on the same file, 11 Oct 2026)
- 13 https://github.com/freriadoc/Avalon/blob/main/README.md (Avalon's README tables: its own runs and Strata v0.2.9 on the same boxes, dated 'Nov 1')
- 14 https://github.com/Indras-Mirror/Xolotl/blob/main/README.md (Xolotl README Gauntlet v3 table and its speed log)
- 15 https://github.com/Indras-Mirror/Xolotl/blob/main/docs/WILL_IT_RUN.md (community runs on small cards)
- 16 https://www.developersdigest.tech/blog/llama-cpp-moe-offload-n-cpu-moe-vs-moe-cache-2026 (write-up of the numbers posted in llama.cpp PRs #29887 and #30112, 9 Oct 2026)
- 17 https://github.com/simplexai-labs/PolyStrata/blob/main/docs/COMMUNITY_BENCHMARKS.md (user-posted runs, including a llama.cpp vs PolyStrata prompt-processing test on two AMD cards)
- 18 https://github.com/Niko1221/Strata/blob/main/docs/COMMUNITY_BENCHMARKS.md (Strata community runs on TITAN RTX, Strix Halo, RX 9070, Tesla P40 and V100 boxes)
- 19 https://github.com/Niko1221/Strata (Strata README table for a 5070, a 3060, one and four Radeon AI PRO R9700, and a Ryzen AI Max+ 395)