Skip to main content
Back

An Independent Researcher Turns Qwen3.8-27B Into a Parallel Decoding Diffusion Model

A researcher converted Qwen3.8-27B into a masked diffusion language model that generates tokens in parallel instead of left to right, with open weights and code.

Model, Full Access
  • Diffusion-Qwen3.8-27B converts Qwen3.8-27B into a masked diffusion LM with parallel unmasking.
  • Causal mask removed from attention; DeltaNet layers add a gated reverse scan initialized to zero.
  • Confidence-threshold decoding: lower thr trades quality for speed.
  • Ships Triton fast path with fused kernels and CUDA graph capture, validated against the original layers.
  • Needs ~56 GB bf16 VRAM; loaded via shipped dqwen.hub, not plain AutoModelForCausalLM.
  • Not benchmarked, no KV cache on single prompts, safety tuning may be weakened.

An independent researcher has released Diffusion-Qwen3.8-27B, an experimental conversion of Qwen3.8-27B from an autoregressive model into a masked diffusion language model. The checkpoint generates multiple token positions during each pass, providing a way to study parallel decoding on an existing 27B model.

The Apache 2.0 repository packages the weights, a custom loader, a fused Triton inference path, a batching engine, and an optional diffusion KV-cache module. In the release post, the author describes it as a research checkpoint intended for experimentation.

Inside the unmasking loop

Diffusion-Qwen generates an output block through repeated denoising passes:

  1. The loader appends a sequence of mask tokens to the prompt.
  2. One forward pass predicts a candidate for every masked position.
  3. Predictions above the confidence threshold are committed and become context for the next pass.
  4. The process repeats until the masks are resolved, then tokens after the first end-of-sequence marker are removed.

Qwen3.8 combines full-attention layers, which perform token-to-token attention, with Gated DeltaNet blocks that carry information through the sequence using a recurrent scan. The conversion removes the causal mask from the full-attention layers, allowing each position to use context from both directions.

Each Gated DeltaNet block also receives a reverse scan. Its output is computed as