An Independent Researcher Turns Qwen3.8-27B Into a Parallel Decoding Diffusion Model
A researcher converted Qwen3.8-27B into a masked diffusion language model that generates tokens in parallel instead of left to right, with open weights and code.
Takeaways
- Diffusion-Qwen3.8-27B converts Qwen3.8-27B into a masked diffusion LM with parallel unmasking.
- Causal mask removed from attention; DeltaNet layers add a gated reverse scan initialized to zero.
- Confidence-threshold decoding: lower
thrtrades quality for speed. - Ships Triton fast path with fused kernels and CUDA graph capture, validated against the original layers.
- Needs ~56 GB bf16 VRAM; loaded via shipped
dqwen.hub, not plainAutoModelForCausalLM. - Not benchmarked, no KV cache on single prompts, safety tuning may be weakened.
An independent researcher has released Diffusion-Qwen3.8-27B, an experimental conversion of Qwen3.8-27B from an autoregressive model into a masked diffusion language model. The checkpoint generates multiple token positions during each pass, providing a way to study parallel decoding on an existing 27B model.
The Apache 2.0 repository packages the weights, a custom loader, a fused Triton inference path, a batching engine, and an optional diffusion KV-cache module. In the release post, the author describes it as a research checkpoint intended for experimentation.
Inside the unmasking loop
Diffusion-Qwen generates an output block through repeated denoising passes:
- The loader appends a sequence of mask tokens to the prompt.
- One forward pass predicts a candidate for every masked position.
- Predictions above the confidence threshold are committed and become context for the next pass.
- The process repeats until the masks are resolved, then tokens after the first end-of-sequence marker are removed.
Qwen3.8 combines full-attention layers, which perform token-to-token attention, with Gated DeltaNet blocks that carry information through the sequence using a recurrent scan. The conversion removes the causal mask from the full-attention layers, allowing each position to use context from both directions.
Each Gated DeltaNet block also receives a reverse scan. Its output is computed as