What 350+ Experiments Taught Us About Serving Qwen 3.6 35B-A3B

NOSIBLE World is a web-scale, point-in-time search and event system. It retrieves dated open-web sources and produces structured records with events, entities, and embeddings. World V1 processed hundreds of billions of text tokens. World V2 will cover a longer history, include more events from categories intentionally dampened in V1 to control volume, and process two to three times as much text. That workload makes the cost of Qwen serving a V2 launch constraint.

We chose the open Qwen 35B-A3B model because no model we assessed delivered higher task quality at lower serving cost. Earlier this year, we benchmarked it against Gemini Flash and DeepSeek for creating NOSIBLE World. Qwen matched or exceeded both on that workload. The benchmark selected the Qwen model family; this campaign selected its serving configuration. We hope Qwen releases a 3.8 35B-A3B variant soon because it would be the first model we test with this serving method for World V2.

Five days and more than 350 serving experiments reduced the steady-state cost of the Qwen 35B-A3B serving branch used to build World from $0.218400/M to $0.125959/M. The final branch generated 6,064.60 completion tokens per second on one RTX PRO 6000 Blackwell, using a $2.75-per-hour GPU-cost normalisation. The campaign covered Qwen 3.5 and Qwen 3.6, and every result retains its actual model label.

1. Checkpoint packages and quantisation

Qwen 3.6 35B-A3B is a mixture-of-experts model with 35 billion total parameters and roughly three billion active parameters per generated token. The checkpoint's routed-expert layout and the kernel that executes it therefore affect serving cost.

We screened weight packages on H200 and RTX PRO 6000 to choose branches for deeper work. The results below use their recorded serving conditions and do not rank every checkpoint for every workload.

NVFP4 uses four-bit floating-point weights. W4A16 uses four-bit weights and 16-bit activations. Those labels omit the execution library, routed-expert layout, auxiliary modules, and available memory that determined these results.

A weight-format label does not predict serving cost. On H200, NVIDIA's ModelOpt NVFP4 Qwen checkpoint reached 6,623.18 completion tokens per second at $0.209701/M; the same path on RTX PRO 6000 reached 2,563.40 tokens per second at $0.297998/M. On the RTX PRO 6000 run, the Marlin path reported no native FP4 compute and used weight-only FP4 compression. Reducing native MTP depth from three proposals to one on H200 improved the package to 7,022.95 tokens per second and $0.197764/M. The FlashInfer TensorRT-LLM mixture-of-experts alternative failed at engine startup on the deployed Blackwell configuration.

Intel's mixed-INT4 AutoRound was the benchmark

Intel's Qwen3.6-35B-A3B-int4-mixed-AutoRound checkpoint kept 243 precision-sensitive modules in FP16 and quantised routed experts to four-bit weights. It loaded, completed the offered client load, and executed on Marlin. It set the Qwen 3.6 cost benchmark during the checkpoint search.

Resident sequences count requests scheduled on the GPU; they exclude clients waiting for work. The explicit-Marlin A/B measured 5,128.61 tokens per second and $0.148947/M, 0.036% below the automatic CUDA 12.9 control. It missed the 0.75% adoption threshold.

Later Intel sweeps produced lower isolated values, including $0.148162/M, but did not replace the replicated automatic control. Experiment 142 remained the canonical Intel configuration. AutoRound documents the quantisation method; the checkpoint card lists the retained modules.

Intel's mixed-INT2 branch required a runtime repair because Qwen's auxiliary predictor contains BF16 tensors while routed experts use the low-bit Humming path. After that repair, two full passes reached 5,218.81 tokens per second and $0.146372/M. The branch did not become the final economic leader. Screen a quantised checkpoint as a package: retained precision, kernel path, auxiliary-module compatibility, and sustained-load behaviour determine the result alongside bit width.

2. Hardware cost

The campaign began on H200 and later screened RTX PRO 6000 Blackwell. Experiment 006 is the Qwen 3.5 campaign starting point. It is outside the Qwen 3.6 GPU comparison. H200 produced more raw completion tokens per second in these records, while RTX PRO 6000 produced the lower cost per completion token.

The H200 and RTX records use different serving conditions and cost bases. They are screening outcomes, not a matched GPU A/B. Calculate cost per completion token with the hourly rate you expect to pay before buying a faster GPU. Include latency targets, replica count, and utilisation when those drive the deployment decision.

3. KV cache and memory capacity

The key-value cache stores context needed to generate the next token. Cache format and available GPU memory therefore determine how much request state the service can retain. In an otherwise identical Intel MTP1/Marlin configuration, FP8 E4M3 key-value cache raised throughput from 4,940.76 to 5,072.34 tokens per second and lowered cost from $0.154610/M to $0.150599/M. Changing its default 16-token pages to 32 tokens reduced throughput to 5,018.22 tokens per second and raised cost to $0.152223/M.

For managed NVFP4, we held the checkpoint and final replay fixture fixed and changed GPU memory available to the runtime.

The 0.96 configuration was 2.10% cheaper than the 0.92 control and 9.81% cheaper than Intel's canonical $0.149000/M result. Moving from 0.92 to 0.94 raised $/M; moving from 0.94 to 0.96 lowered $/M. The final path used FP8 E4M3 cache, an eight-bit float with four exponent bits and three mantissa bits.

4. Admission control and scheduler policy

Admission and scheduling controls set the operating limits:

• Reducing resident capacity from 512 to 480 lowered $/M by 0.48%. The gain missed the adoption threshold, so 512 remained the operating setting.
• At 1,536 offered clients, $/M rose by 0.91% and mean time to first token increased by 3.76 seconds. At 1,024 offered clients, $/M rose by 0.44% and mean time to first token fell by 3.81 seconds. The 1,280-client setting remained the cost choice for this batch service.
• Reducing the model-length ceiling from 2,048 to 1,024 tokens produced 5,110.34 tokens per second and $0.149479/M in the Intel control. It was 0.32% slower and more expensive, and it excluded World inputs that span 1,024 to 2,048 tokens.
• Batch-token limits of 12,288 and 20,480 cost $0.135746/M and $0.135406/M, respectively, versus $0.134386/M at 16,384.
• A 24,576-token setting failed the warm-up gate with a CUDA out-of-memory error before a full score could exist.
• A dense CUDA Graph ladder shaped around observed traffic was 3.42% slower than the standard ladder.
• Disabling prefix caching, chunked prefill, or statistics failed to clear the 0.75% adoption threshold. Requests for alternate asynchronous scheduling produced no material gain, but logs did not confirm that the alternate scheduler activated.

Cache format, memory fraction, resident capacity, batch capacity, prefill policy, and graph capture compete for GPU memory and scheduling time. Longer prompts increase cache pressure. Longer completions change the prefill/decode mix. Latency-sensitive services can prefer a capacity setting that a saturated batch service rejects. Select the cheapest setting that completes the required workload.

5. Native MTP and NTP

Standard next-token prediction (NTP) generates one token per decoding step. Qwen's native multi-token prediction (MTP) path adds a predictor that proposes tokens for the full model to accept or reject. We tested the NTP baseline, MTP with one, two, and three proposals, and a language-model-only MTP control. Accepted proposals must save more target-model work than the predictor adds in compute and memory overhead.

Removing the MTP predictor (the NTP baseline) reduced matched-control throughput by 11.04%. In the earlier configuration with 0.92 GPU-memory utilisation and a 24,576-token batch limit, one MTP proposal cost $0.151559/M and two proposals cost $0.160839/M. The second proposal raised cost. In an otherwise matched Intel test, three proposals reached 4,534.37 tokens per second: 6.91% slower and 7.43% more expensive than one proposal.

6. DFlash: not viable for World

DFlash is a separate block-diffusion speculative-decoding method. It uses the official FP8 target and an external eight-token draft. It does not use Qwen's native MTP head. We tested it as a separate serving mode on SGLang.

The valid H200 run queued 1,280 offered clients behind 32 resident requests. It delivered a real full-load result, but at $0.387549/M it was not cost-competitive for World. No no-draft control used the same SGLang runtime and 32-resident shape, so this result rejects the deployment configuration; it does not isolate DFlash overhead. The high-capacity H200 route and every RTX route stopped at compatibility or startup gates. DFlash did not enter the deployment choice.

Draft acceptance, draft-state memory, target-GPU kernel support, and scheduler capacity can each eliminate a net speculation gain. Compare each separate draft method with a matched no-speculation control.

7. Execution paths

A kernel is the GPU program that performs a model operation. Marlin, FlashInfer, CUTLASS, Triton, TensorRT-LLM, and DeepGEMM provide low-precision execution paths. A runtime can select a path, fall back to another, or reject a layout.

In a controlled official Qwen 3.5 FP8 H200 test, enabling DeepGEMM delivered 7,197.30 completion tokens per second and $0.192974/M. Disabling only DeepGEMM delivered 7,661.29 tokens per second and $0.181286/M: 6.45% more throughput and 6.06% lower cost. The kernel remained compatible; it was slower for this workload.

B12X mixture-of-experts routing

B12X was a separate FlashInfer execution route for the NVFP4 target's mixture-of-experts layers. It selected successfully and completed World traffic, but it was expensive in every early configuration.

Later source tests could select B12X but could not carry the full World load. One stopped before API readiness, two could not allocate a 2.58 GiB minimum cache, a reduced-batch warm-up timed out 168 of 1,024 clients, and the first scored pass completed 1,520 of 4,096 requests. B12X did not advance.

Unsloth NVFP4 began at 5,040.22 tokens per second and $0.151559/M, behind Intel. Memory and scheduler changes lifted managed NVFP4 to 5,684.28 tokens per second and $0.134386/M. We ran the source runtime in fresh allocations and averaged two passes at 6,064.60 tokens per second and $0.125959/M with the same checkpoint.

The final runtime used vLLM source commit 226749b, built with CUDA 13 for Blackwell. Startup logs confirmed the selected path:

• FlashInfer CUTLASS for NVFP4 linear operations.
• FlashInfer CUTLASS for NVFP4 mixture-of-experts operations.
• FlashInfer attention.
• FlashInfer CUTLASS for the unquantised auxiliary predictor.

The source runtime used Triton kernels v3.5.1 and a Blackwell adaptation that disabled an unsupported FlashAttention-3 (FA3) extension. The adaptation changes the build and selected attention path. Runtime logs selected FlashInfer attention.

Alternate execution paths either rejected the NVFP4 checkpoint, failed the 1,280-client load, or cost more. An exact-shape mixture-of-experts autotuning test changed the maximum graph-capture size for FlashInfer CUTLASS autotuning. It completed at 5,990.35 tokens per second and $0.127520/M, 1.24% more expensive than the automatic final path. The source winner completed two full fresh-allocation passes.

This result was 6.27% cheaper than managed NVFP4 and 15.46% cheaper than Intel's canonical configuration. We selected the automatic backend after reviewing startup logs and reproducing the result in a fresh allocation.

8. Transport and release gates

Delivery mode and client concurrency can change a serving result under offered load. No delivery variant produced a repeatable performance gain, so none entered the final configuration.

Managed NVFP4 completed an endurance run of 16,384 requests without HTTP or streaming errors. Its deterministic translation screen covered 16 source languages and found no blank output, repetition loop, malformed response, or observed systematic omission of named entities or numbers. The source runtime has throughput evidence, not output qualification. It must pass the same output screen before promotion.

Release decisions

Test protocol and adoption rules

In the matched final RTX fixture, Qwen 3.5 Intel AutoRound cost $0.149374/M and Qwen 3.6 Intel AutoRound cost $0.149000/M. The 0.25% difference missed the 0.75% adoption threshold, so we treat the releases as one serving family for this non-thinking workload.

Each experiment answered a bounded question: whether a checkpoint loaded, whether a kernel supported its layout, whether a serving shape survived warm-up, or whether every client request completed. A failed gate ended that branch before it consumed full-load test time. The final Blackwell comparison records replayed NOSIBLE World translation requests with a 2,048-token context limit, streaming output, disabled thinking, a 256-token completion limit, 1,280 offered clients, and two measured 4,096-request passes.

Every request had to finish. Earlier feasibility, package, and H200 screens chose candidates. They were not matched final comparisons. The final result applies to this replay fixture. Different GPU pricing, prompt lengths, completion lengths, offered concurrency, latency limits, or quality requirements require rerunning the relevant tests.

The campaign used five gates:

1. A functional request confirmed a valid response.
2. A high-concurrency warm-up exposed capacity and stability problems.
3. Two full passes established throughput and cost before a late-stage candidate could change the recommendation.
4. A fresh allocation reproduced material gains.
5. Endurance and output checks qualified a release candidate.

The headline reports normalised steady-state GPU cost:

normalised cost per million completion tokens =
    hourly GPU cost x 1,000,000 / (completion tokens per second x 3,600)

The final RTX comparison uses $2.75 per hour as a campaign normalisation. Experiment 332 used an allocation priced at $2.09 per GPU hour plus $0.021 per disk hour, so $0.125959/M is not a provider-neutral deployment quote. H200 records retain their recorded $5-per-hour rate. The calculation excludes startup, source compilation, model download, and JIT work; short-lived jobs must amortise those costs separately.

We adopted only replicated improvements of at least 0.75%. An apparent gain above 3% required a fresh-allocation reproduction. Smaller measured differences stayed in the record but did not change the configuration.

Measurement gates for the Qwen serving campaign

The decision map

Decision ledger for the major Qwen serving choices

Actions for another serving campaign

1. Choose GPUs based on output cost, latency, capacity, and quality requirements.
2. Screen checkpoint packages before deep parameter sweeps; a mixed-precision checkpoint can beat a lower-bit format.
3. Tune cache format, memory fraction, batch limit, resident capacity, and graph capture together because they use the same GPU memory and scheduling resources.
4. Compare native MTP with its NTP control, then test external draft methods as separate serving modes.
5. Record the runtime path and build adaptations because kernel selection can change measured throughput and $/M.

For the fixed NOSIBLE World replay fixture, Unsloth NVFP4 Fast on RTX PRO 6000 Blackwell was the steady-state performance leader at 6,064.60 completion tokens per second and $0.125959/M. The source runtime used one native proposal, FP8 E4M3 cache, 0.96 GPU-memory utilisation, 16,384 batch tokens, and 512 resident sequences. The source runtime must pass the retained output screen before release.

World V2 is why we ran this campaign. Re-run the relevant branches when GPU architecture, GPU price, prompt length, completion length, offered concurrency, latency target, or quality requirements change. For a new Blackwell stack, start with the NVIDIA RTX PRO 6000 specifications, the CUDA compatibility guide, and the NVIDIA Developer Forums, then measure the service.

External sources and implementation terrain

• Qwen 3.5 35B-A3B FP8 model card and Qwen 3.6 35B-A3B FP8 model card
• NVIDIA Qwen 3.6 NVFP4 checkpoint, Intel mixed-INT4 AutoRound checkpoint, and Unsloth NVFP4 Fast checkpoint
• NVIDIA Model Optimizer NVFP4 reference, AutoRound, and Marlin
• vLLM, vLLM speculative decoding, vLLM FP8 key-value cache, FlashInfer, CUTLASS, Triton, TensorRT-LLM, and DeepGEMM
• DFlash, NVIDIA H200, NVIDIA RTX PRO 6000 Blackwell, Gemini models, DeepSeek, and the NVIDIA Developer Forums