Qwen3.8-27B · RTX 5090 Runtime Recipe

Can a dense 27B model run at interactive speed on a single 32GB RTX 5090, and which runtime is actually best for a given workload? The second question matters because raw decode throughput is not the same as agent throughput — a runtime that wins a synthetic benchmark can lose on a real autonomous trajectory.

The setup uses one workstation GPU: an RTX 5090 reporting 32,607 MiB of VRAM. One generation runtime and one request run on the GPU at a time. All serving is loopback-only (127.0.0.1), with no external cloud API dependency and no modified model weights in the repository.

Three pinned inference backends sit behind a workload-aware task router that stops and cleans up the previous runtime before starting exactly one loopback backend by workload class: llama.cpp Q5_K_M + MTP3 as the default single-agent runtime, SGLang NVFP4 as the high-concurrency serving baseline, and NInfer NVFP4 + FP8 KV + MTP3 as a qualification candidate.

Measured values are tied to their runtime, artifact, context, and workload. SGLang NVFP4 records ~69.3 tok/s on short context and ~60.8 tok/s at 80K+ (median of 43 samples). llama.cpp Q5_K_M + MTP3 records 151.72 / 120.29 / 98.59 / 94.66 tok/s across short → 32K → 80K → 113.9K context. On a real 100K+ autonomous coding trajectory (CourseBench long task, 2026-08-19 re-qualification), Q5_K_M + MTP3 averaged 109.51 tok/s at a max observed context of 106,829 tokens, with 89.61% speculative acceptance and 28.63 GB peak VRAM.

The data points to different strengths rather than one universal winner. Q5/MTP3 decodes roughly 1.5–2× faster in bounded runs, while SGLang provides stable concurrent serving at 80K+ context. Both runtimes passed the correctness gates: basic 6/6, JSON schema 20/20, tool calls 40/40, NIAH at 32K/80K/113K+ 5/5 each, with zero CUDA/OOM/output-corruption events. Raw decode TPS therefore does not represent agent throughput on its own.

The final production speed ranking awaits comparable end-to-end wall-time measurements across multiple seeds, a limitation documented in the repo. The September 2026 NInfer comparison is complete and a Codex failure is documented; Q5 remains the default single-agent runtime while the wall-time comparison is pending.