Qwen3.8-27B-NVFP4-RTX5090

Runs on **SparkInfer, SGLang and vLLM** unmodified — configs for all three are [below](#serving). Serving many users at once? See [concurrency](#serving-more-than-one-user). **SparkInfer × this NVFP4 build × the DSpark v2 drafter** — an engine, a checkpoint, and a speculative drafter optimized against each other, compounding to **4.3×**. The drafter never changes what the model says: the target verifies every drafted token. **GeForce RTX 5090–specific** NVFP4 checkpoint of [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B), quantized with [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer). Serves the **full native 262,144-token context** on 32 GB. With the [DSpark v2 drafter](https://huggingface.co/gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4): **264.8 tok/s overall — up to 420 on code — on [SparkInfer](https://github.com/gittensor-ai-lab/sparkinfer)** (its bench harness; the HTTP server is autoregressive-only today) and **161.7 tok/s on SGLang's OpenAI server**. Without speculation: 92.9 tok/s SparkInfer · 85.8 SGLang. **This is the final build.** Everything is quantized down to the `lm_head`, the unused MTP head is **removed from the weights**, and the file ships as two shards: Blackwell tensor cores only. Hopper can load the files but cannot run NVFP4. [**SparkInfer**](https://github.com/gittensor-ai-lab/sparkinfer) is a zero-dependency C++/CUDA engine for Blackwell. The image is prebuilt and published to GHCR — no compiler, no CUDA toolkit, no Python stack: docker run --gpus all -p 8080:8080 -v qwen38:/models \ ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:latest **These weights download themselves on first run** into the `qwen38` volume (~18 GB, cached after), so that one command is the whole setup. The server is ready in about 10 s on later starts. curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "qwen38-nvfp4", "messages": [{"role": "user", "content": "What is the capital of Japan?"}], "chat_template_kwargs": {"enable_thinking": false} }' Serves **text, images and video** through the same OpenAI-compatible API. ~1 GB image, Blackwell (`sm_120`) only. One caveat if you wire this into an agent framework: `/v1/models` currently advertises `input_modalities` as text and image only, so a client that routes on that field will not offer video even though `video_url` parts work — send video explicitly for now ([sparkinfer#983](https://github.com/gittensor-ai-lab/sparkinfer/issues/983)). Provenance is attested to the image digest: gh attestation verify oci://ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:latest \ -R gittensor-ai-lab/sparkinfer Defaults are the values measured as correct on a 32 GB card, so the two easy mistakes are pre-solved. Override any of them with `-e`: The output cap reserves context per request, so raising it without also raising `CTX` makes concurrent requests fail with *server overloaded*. The NVFP4 `lm_head` loads natively — nothing needs converting. The same image runs the DSpark benchmark, fetching the drafter on first use: docker run --rm --gpus all --ipc=host -v qwen38:/models \ ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:latest bench code git clone https://github.com/gittensor-ai-lab/sparkinfer && cd sparkinfer cmake -B build -DCMAKE_CUDA_ARCHITECTURES=120 -DBUILD_SERVER=ON && cmake --build build -j ./build/server/sparkinfer_server -m <weights> --tokenizer <weights>/tokenizer.json \ --ctx 262144 --host 0.0.0.0 --port 8080 Needs CUDA 12.8+ and rustc ≥ 1.79. 24 prompts across four workloads (chat, code, math, JSON), 256 max tokens, `temperature=0`, thinking off, sequential, through each engine's OpenAI API, every row measured at `--ctx 262144`. SGLang 0.5.18 with FP8 KV; SparkInfer v0.5.5 with its default int8 KV. SparkInfer is both the faster engine without speculation and the one that holds the most context: 262,144 tokens costs it 27.9 GB, and it still answers at `--ctx 360000` (31.2 GB), though decode falls to ~74 tok/s at that extreme. Three published NVFP4 checkpoints of Qwen3.8-27B, run on **one RTX 5090** — same engine (vLLM), same session, same sampling (the model's own `generation_config`: temp 1.0, top_k 20, top_p 0.95), same 32,768-token budget. **Quality is a tie; the difference is what the weights leave room for.** All three confidence intervals overlap, truncation was comparable (41 / 45 / 42), and long-context reasoning is a dead heat. This build is **24.9% faster with 43% more context** than NVIDIA's official checkpoint. The reason is allocation, not luck. NVIDIA keeps self-attention and Gated-DeltaNet at **FP8** — 22% of the linear weight — which costs both decode bandwidth and KV headroom on this card; their card targets **GB300**, where FP8 sits much closer to FP4 in relative cost. QUASAR keeps `lm_head` in BF16 (2.54 GB against our 0.71 GB), read on every token. We quantize both, and spend the difference on context. Measured on the same box: leaving NVFP4 for the next instruction tier costs **1.5–2.1×** on these GEMM shapes, so precision spent elsewhere is expensive here. > Two caveats. Published BF16 references disagree across harnesses (NVIDIA reports 88.92 on GPQA > Diamond, QUASAR 91.41, for the same weights) — so only the same-harness numbers above should be > compared with each other, never against a published leaderboard. AA-LCR here is scored with a > deterministic salient-token proxy applied identically to all three, not Artificial Analysis' > LLM judge; it ranks these builds but is not comparable to their published figures. Every figure above is a single stream. Under concurrent load SparkInfer v0.5.5 decodes the whole batch in one packed forward — earlier versions time-sliced one sequence at a time and stayed flat at ~77 tok/s no matter how many clients arrived. Distinct prompts per request so no cache flatters the result, 128 tokens each, `--ctx 40960`, measured client-side over the streaming API. 32/32 requests succeeded at every level. Aggregate throughput peaks around 8–16 concurrent and is flat to 32; per-request latency degrades from there, so 8 is the sweet spot on one card. **Where this loses.** vLLM 0.28 on the same weights, same box, same harness: 134.5 / 266.6 / 545.8 / 546.6 tok/s at 2 / 4 / 8 / 16. We lead at 2 concurrent and trail by about 1.6× at 8–16. If you are serving many simultaneous users, vLLM is still the faster server; if you are serving one stream, or a few, SparkInfer is the faster engine and holds far more context. A configured prefix — a long system prompt, a document, a tool manifest — is kept across requests instead of being re-prefilled. With a 32,022-token shared prefix: Change one word of the prefix and the next request costs 3.30 s again, which is what makes the speedup attributable to the cache rather than to warm-up. For "load a document, ask ten questions", or agentic traffic that repeats a system prompt and tool definitions on every call, this outweighs any decode difference on this page. This is a **configured** prefix (`SPARKINFER_SERVER_PREFIX_TOKEN_FILE`, a file of token ids), not automatic caching over whatever prefix your requests happen to share, and it engages on sequential requests rather than under concurrency. SparkInfer's DSpark path exists in its **bench harness**, not yet in the HTTP server — so it is not something you can `curl` today, but it does show what the engine and this drafter do together. Measured on the same checkpoint and the same [DSpark v2 drafter](https://huggingface.co/gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4), 256 tokens per workload, greedy: Against SGLang's **1.89×** on the same drafter, this is the larger speculation gain — the block drafter feeds a verifier that keeps up with it. Code accepts nearly the whole 8-token block (τ 6.78 of a possible 8). **Read these as their own harness, not as server numbers.** The bench runs at a few hundred tokens of context, where its autoregressive baseline is 97.1 tok/s against the server's 92.9, and it uses one prompt per workload rather than the server benchmark's six. What transfers is the ratio, not the absolute tok/s. Speculation is lossless by construction — the target verifies every drafted token — and 9 of 12 bench runs came back token-identical to their autoregressive baseline. The three that differed diverged at a single token and then followed a different but valid continuation, which is what a near-tie in the target's own logits looks like when the verify step uses a different GEMM shape than single-token decode. If you want speculation over HTTP today, serve with SGLang. This build and RadixArk were measured here back to back on identical SGLang flags (FP8 KV, `--mm-feature-transport cpu`, 2 running requests), both at `--ctx 262144`. Unsloth's figures are the earlier vLLM measurement on the same GPU — a different engine, included for scale. Two gaps stand out. RadixArk's drafter is BF16 (3.5 GB against our 1.41 GB), and its draft KV pool is sized to the full target context, so turning speculation on costs it most of its window: **37,900 tokens left**, where ours keeps 165,169. And RadixArk **does not load on SparkInfer at all** — its Gated-DeltaNet projections (`linear_attn.in_proj_qkv`, `in_proj_z`, `out_proj`) are rejected by the compressed-tensors loader as malformed FP8. This checkpoint runs on all three engines unmodified. Quantizing the `lm_head` and dropping the MTP head did not move quality. On a matched 60-item smoke (GPQA Diamond, AIME 2025, MMLU-Pro — 20 items each, thinking on, `temperature=1.0`, seed 20260815): The two engines agree on the weights. The budgets differ because SparkInfer caps output per request, so read this as *same weights, same accuracy* — not as an engine ranking. Both runs carry 6–9 length truncations per task, where thinking filled the budget before an answer appeared. For a like-for-like comparison against Unsloth NVFP4, the earlier vLLM measurement on the previous build of this checkpoint stands at **45/60 (75%) for both**: GPQA 13/20 vs 14/20, AIME 15/20 vs 14/20, MMLU-Pro 17/20 tied. **These are 20-item smokes, not published benchmark scores.** Do not cite them as GPQA / AIME / MMLU-Pro results. docker run -d --name sgl --gpus all --ipc=host --shm-size 32g -p 30000:30000 \ -v hf_cache:/root/.cache/huggingface \ lmsysorg/sglang:latest \ sglang serve \ --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \ --trust-remote-code --tp-size 1 \ --context-length 262144 \ --kv-cache-dtype fp8_e4m3 \ --attention-backend flashinfer \ --chunked-prefill-size 2048 \ --mamba-radix-cache-strategy extra_buffer_lazy \ --mamba-ssm-dtype bfloat16 \ --mem-fraction-static 0.90 \ --max-running-requests 2 \ --max-mamba-cache-size 12 \ --speculative-algorithm DSPARK \ --speculative-draft-model-path gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4 \ --speculative-dspark-block-size 7 \ --speculative-draft-model-quantization modelopt_fp4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --host 0.0.0.0 --port 30000 Drop the four `--speculative-*` lines for the maximum-context profile. Two flags that are easy to get wrong: - **`--speculative-draft-model-quantization modelopt_fp4`** is required. Without it SGLang assumes BF16 and mis-loads the packed U8 tensors. - **`--max-mamba-cache-size`** must leave headroom. Under DSpark each request takes **4** Gated-DeltaNet state slots, so size 4 provisions exactly one request with nothing spare and eviction can fail with `AssertionError: Can not alloc mamba cache`. Rule of thumb: **≥ 4 × `--max-running-requests` + 4**. vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \ --quantization modelopt \ --kv-cache-dtype fp8 \ --trust-remote-code \ --max-model-len 262144 \ --max-num-seqs 16 \ --gpu-memory-utilization 0.97 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml vLLM **0.27.x**. First boot JITs the FlashInfer SM120 FP4 GEMM (`nvcc` + CUDA 13 headers); limit parallel JIT on smaller host RAM with `MAX_JOBS=2`. Util **0.97** is required for native 256k on 32 GB — 0.90 only holds ~205k KV. Qwen3.8 is a thinking model; for short answers pass `"chat_template_kwargs": {"enable_thinking": false}`. Qwen3.8's own template, with agentic fixes. It renders **byte-identical to upstream `Qwen/Qwen3.8-27B` on every non-tool path**, so ordinary chat, thinking and vision behave exactly as the base model does. `xhigh` (default), `medium`, `low` — invalid values raise: {"chat_template_kwargs": {"reasoning_effort": "medium"}} On this checkpoint **`xhigh` is the cheapest setting at equal accuracy.** It reasons more directly rather than simply longer: 12 verifiable problems, `temperature=0`, 3k budget, no truncations. A harder 10-problem set held the same ordering (562 vs 939 tokens). Keep the default unless you have a specific reason not to. XML by default, matching the `qwen3_xml` (vLLM) and `qwen3_coder` (SGLang) parsers: Paris Assistant `tool_calls` replayed from an OpenAI-style response render whether `function.arguments` is a **dict or a JSON string**. The string form previously raised `TypeError: Can only get item pairs from a mapping` on the second turn of an agentic loop, which broke direct `apply_chat_template`, llama.cpp and LM Studio users. Server-side tool calling was never affected — SGLang and vLLM normalize arguments to a dict before rendering.