protoLabsAI's Ornith-1.0 Hits 1.7x Faster Decoding With One Flag
A grafted multi-token prediction head turns Ornith-1.0-9B into a self-speculative decoder in llama.cpp, cutting local latency by up to 1.73x with no quality loss.
Takeaways
- protoLabsAI shipped GGUF builds of Ornith-1.0-9B with a KL-distilled MTP head baked in for llama.cpp self-speculative decoding.
- Measured 1.4x to 1.73x decode speedup on RTX A6000, distribution-lossless, no separate draft model needed.
- Novel finding: on Blackwell, NVFP4 quant beats Q4_K_M by ~28% because MTP verify is nearly free on FP4 tensor cores.
- Per-token acceptance of 0.766 in llama.cpp matches the vLLM reference (0.762) for the same head.
- Base is DeepReinforce's Ornith-1.0-9B, a Qwen3.5-based agentic coding model scoring 69.4 on SWE-Bench Verified.
- Requires llama.cpp b9616+ with
--spec-type draft-mtp; recipe and standalone head released under MIT.
Speculative decoding usually means running two models side by side: a small draft model that guesses ahead and a large target model that verifies. protoLabsAI just shipped GGUF builds of Ornith-1.0-9B that fold that trick into a single file. The draft head is grafted directly onto the trunk, so llama.cpp handles multi-token self-speculation with one flag and no second checkpoint to load.
The measured payoff on an RTX A6000 is a 1.4 to 1.7x single-stream decode speedup, and it comes without changing the output distribution. Every file carries the nextn head, so the --spec-type draft-mtp flag just works. If you already run Ornith locally through llama.cpp, LM Studio, or Ollama, this is a drop-in replacement.
How the draft head got there
Multi-token prediction (MTP) trains an extra small head that predicts the next N tokens in parallel. Those predictions become the draft, and the main model verifies them in a single batched forward pass. Because acceptance is rejection sampling against the target, the head is trained with KL divergence to the target's own next-token distribution rather than hard cross-entropy, which would sharpen the argmax but miscalibrate the distribution.
The head itself was initialized from Qwen3.5-9B's mtp.* tensors, then re-trained on Ornith-9B's own generations. The recipe is open: