Alibaba's Qwen3.8 Runs at 112 tok/s on a $949 Intel GPU
A community 4-bit quant of Qwen3.8-27B keeps the multi-token prediction head in BF16, hitting 112 tokens per second on a single Intel Arc Pro B70.
Takeaways
- Community GPTQ INT4 quant of Qwen3.8-27B keeps the MTP draft head in BF16 for native speculative decoding.
- Hits 83.7 tok/s baseline and 112.65 tok/s with an optional INT4 draft overlay on one Intel Arc Pro B70.
- MTP acceptance 95.9% BF16 draft, 94.4% with INT4 overlay; 12/15 on a temperature-0 task suite.
- Five concurrent 8K coding sessions serve 25.5 tok/s each with a required
gdn_attentionmixed-split patch. - Ships full F16 vision tower plus preprocessor configs; image+text request finishes in 7.3 seconds.
- Apache-2.0, deployable via vLLM XPU or SGLang; recipe in the Intel Arc Pro B70 Inference Cookbook.
A community-quantized build of Qwen3.8-27B has landed on Hugging Face with a specific goal: squeeze Alibaba's new dense multimodal model onto a single $949 Intel Arc Pro B70 while keeping native speculative decoding fully functional. The SergiioB checkpoint is a GPTQ INT4 conversion of the base model where the multi-token prediction draft layer is deliberately left in BF16, so the model can draft its own speculative tokens without a second checkpoint in VRAM.
Alibaba describes Qwen 3.8 27B as a native multimodal dense model with 262K native context extensible to 1M tokens via YaRN, and the model card confirms a Causal Language Model with a Vision Encoder at 27B parameters. Architecturally it uses a 5120 hidden dimension across 64 layers with a hybrid layout of 16 repeats of three Gated DeltaNet-plus-FFN blocks followed by one Gated Attention-plus-FFN block, and it ships with Multi-Token Prediction trained across multiple steps that downstream quants exploit for speculative decoding.
Don't quantize the drafter
Multi-Token Prediction (MTP) is a form of self-speculative decoding. Its main practical advantage over runtime speculative decoding is memory: EAGLE-3 and DFlash require a separate draft checkpoint loaded alongside the target model, and at high batch sizes that draft model's memory footprint competes with the target's KV cache. MTP heads share the main model's KV cache without a separate KV structure because the future-token heads read from the same key-value tensors that the trunk already computed, so no additional KV pages are allocated per request.