Skip to main content
Back

Ornith's DFlash Drafter Pushes Ornith-1.5-35B Decoding to 2.32× Faster

Ornith pairs its 35B mixture-of-experts reasoning model with a block-diffusion draft that pushes decoding past 180 tokens per second on a single A6000.

  • Ornith released a block-diffusion draft model for speculative decoding on its 35B MoE reasoning model.
  • Community benchmark shows 2.32x decode speedup on two A100s with GSM8K and HumanEval quality preserved.
  • AWQ-quantized target plus DFlash2 hits roughly 180 tokens per second on a single RTX A6000.
  • The drafter proposes whole token blocks in parallel via diffusion, then the 35B target verifies them losslessly.
  • Ships with validated vLLM and SGLang recipes; requires SGLang 0.5.18 or vLLM 0.20.2 and up.
  • Not useful on small consumer GPUs where the fast MoE baseline already beats draft-plus-verify overhead.

Ornith has released a DFlash checkpoint that accelerates Ornith-1.5-35B-A3B by proposing blocks of tokens in parallel. The 35B target model scores those proposals and accepts valid tokens before generation continues. Fewer target-model passes can raise decode throughput at the low batch sizes common in coding agents and interactive reasoning workloads.

A separate single-GPU demo reported about 180 generated tokens per second on an RTX A6000 with 48 GB of memory. That configuration combined an AWQ-quantized target with the newer DFlash2 drafter, which adds path selection and a two-tap dynamic convolution. Its hardware, target precision, and drafter differ from the stock DFlash benchmark.

The target keeps control

The DFlash checkpoint serves only as a companion to Ornith-1.5-35B-A3B, which performs verification and produces the final output. The target is a mixture-of-experts model with 35B total parameters and about 3B active parameters per token. Sparse activation reduces computation, although all 35B parameters must remain available in memory.

During decoding, DFlash proposes several token positions at once and the target scores the block in parallel. Exact acceptance and rejection logic preserves the target model’s probability distribution. Randomized runs can still produce different text because distributional equivalence does not guarantee byte-for-byte output under sampling.

Diffusion fills the block

Conventional speculative drafters, including EAGLE-style heads, extend candidate sequences token by token. DFlash initializes an entire block and refines its positions through a short sequence of parallel denoising steps before verification. The approach aims to produce more useful candidate tokens per draft operation.