Tencent's DSpine Pushes Qwen3.8-27B to 104.8 tok/s on a Single GPU
A community GGUF port brings Tencent's DSpine draft model to llama.cpp, pushing Qwen3.8-27B decoding to ~105 tok/s on a single RTX 3090.
Takeaways
- GGUF port of Tencent's DSpine draft model for Qwen3.8-27B is live on Hugging Face.
- Hits ~105 tok/s on one RTX 3090 vs 58 for DFlash2 and 29 without speculation.
- Five-layer drafter produces 16 tokens in parallel with causal block attention and a per-layer code bridge.
- Accepts ~9.5 tokens per verify step on math, ~8.9 on C++, only ~2.8 on prose.
- Requires the author's dspine fork of llama.cpp; upstream support is not landed.
- Apache-2.0 licensed; Q4_K_M file is 1.26 GB on top of the Qwen3.8-27B target.
A community llama.cpp port of Tencent’s DSpine speculative-decoding model reached 104.8 decode tokens per second with Qwen3.8-27B in the author’s single-GPU test. The same target model produced approximately 29 tok/s without speculation, giving DSpine a 3.6x advantage. It also outpaced the author’s DFlash2 result of 58.2 tok/s.
The release includes GGUF files converted from Tencent’s original checkpoint. The 2-billion-parameter drafter pairs with the full Qwen3.8-27B target and carries an Apache-2.0 license, which permits commercial use subject to the license terms.
How DSpine drafts 16 tokens
Speculative decoding reduces target-model work by asking a smaller model to propose several upcoming tokens. The target evaluates the proposed block in one forward pass and accepts the valid prefix. A higher acceptance rate spreads each expensive target pass across more generated tokens. With compatible sampling and verification, final token selection continues to follow the target model’s decoding rules.
DSpine builds on Tencent’s earlier DFlash and DFlash2 designs. Its five-layer draft network reads hidden states from five target-model layers and predicts a 16-token block in parallel. Three mechanisms help those parallel predictions retain information about preceding tokens:
- Causal block attention: Each draft position can attend only to earlier positions in the same block, preventing access to future tokens.
- Code bridges: After every layer, each position receives a gated message derived from its predecessor’s predicted token code. This carries causal information through the draft network’s depth.
- Serial last-write refinement: For the top 16 candidates at each position, the drafter recalculates the final bridge message using each possible predecessor’s exact code. The resulting 16-by-16 transition table lets the decoder select a coherent chain through greedy or probabilistic sampling.
These mechanisms increase the number of draft tokens accepted during each target-model verification, with the largest gains appearing on structured output such as code and mathematics.
The numbers on one RTX 3090
The author measured throughput on a single 24 GB RTX 3090 using a Q4_K_M target, llama-server, one processing slot, and prompts configured to generate as many as 768 new tokens.
| Configuration | Decode tok/s | Tokens per verification |
|---|---|---|
| Target only | 28.9 | 1.0 |
| DFlash2 Q4_K_M, 7 draft tokens | 58.2 | About 5.1 |
| DSpine Q4_K_M, 15 draft tokens | 104.8 | About 6.3 |
| DSpine Q8_0, 15 draft tokens | 102.2 | About 6.3 |
A separate DSpine Q4_K_M run reached 104.1 tok/s at temperature 1.0 with probabilistic draft sampling. Its throughput remained close to the headline result even after sampling introduced more variation into the generated sequence. These measurements come from the conversion author and have not been independently reproduced here.
Structured output gets the largest gains
Acceptance varied substantially by workload in the author’s tests:
- Math: about 9.5 tokens per verification step.
- C++: about 8.9 tokens per verification step.
- Python: about 7 tokens per verification step.
- Prose: about 2.8 tokens per verification step.
Constrained syntax, repeated patterns, and predictable continuations give the drafter more opportunities to match the target. Free-form prose presents more plausible next-token choices, which lowers acceptance and reduces the speedup.
Longer prompts also reduced performance. With a 13,000-token prompt, which exceeded the drafter’s 8,192-token context window, the author measured 62 tok/s and about 4.1 tokens per verification step. The configuration continued to accelerate generation, though its throughput fell well below the short-context peak.
Build the fork and choose a quant
Upstream llama.cpp does not currently implement DSpine. The converted files require the author’s DSpine branch, while stock builds reject them during loading.
| Quantization | File size | Intended use |
|---|---|---|
| Q4_K_M | 1.26 GB | Recommended runtime file; code-bridge tensors remain in BF16. |
| Q8_0 | 2.20 GB | Higher-precision runtime file; it matched Q4_K_M acceptance in the author’s tests. |
| BF16 | 4.06 GB | Full-precision source for creating other quantizations. |
Compile the CUDA build
git clone --branch dspine https://github.com/Ajay9o9/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
Start a single-slot server
./build/bin/llama-server \
-m Qwen3.8-27B-Q4_K_M.gguf \
-md Qwen3.8-27B-DSpine-b16-Q4_K_M.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 15 \
-ngl 999 --spec-draft-ngl 999 \
-fa on --jinja -c 32768 -np 1
-mselects the Qwen3.8-27B target GGUF.-mdselects the DSpine draft GGUF.--spec-type draft-dflashactivates the speculative backend used by the port.--spec-draft-n-max 15allows the server to use nearly the full 16-token draft block.-ngl 999and--spec-draft-ngl 999request GPU offload for both models.-np 1runs one processing slot, matching the headline benchmark configuration.
The default --spec-draft-n-max value is 3, which leaves most of DSpine’s draft block unused and sharply limits throughput. A value of 15 produced the best reported single-slot result.
Two concurrent slots require a smaller draft block on a 24 GB card. The author recommends --spec-draft-n-max 7 with -np 2 because Qwen3.8’s hybrid architecture stores recurrent state for every token under verification, and two 15-token slots exceed the available memory.
What developers gain
A measured 104.8 tok/s gives a local 27B model enough decoding throughput for low-latency code generation, agent loops, retrieval-augmented generation, and chat on one RTX 3090. The workload breakdown sets practical expectations: code and math receive the largest gains, while prose benefits less from the drafter.
DSpine changes inference cost without modifying the 27B target’s weights. Its immediate constraint is deployment support, since users must maintain a custom llama.cpp build. Upstream integration would remove that requirement and make the converted draft model usable through standard llama.cpp releases.