Skip to main content
Back

Qwen3 Thinks 22x Faster With a 20-Line Open-Source Grammar Fix

A 20-line grammar file cuts Qwen3.6 thinking tokens by 22x on HumanEval+ with zero fine-tuning, and even lifts pass@1 on LiveCodeBench.

  • Repo structured-cot constrains only the reasoning block via a GBNF grammar file.
  • On HumanEval+: 92.7% vs 92.1% pass@1 with 22x fewer thinking tokens (138 vs 3087).
  • On LiveCodeBench v6 LeetCode slice: +14 pp pass@1 (64% vs 50%) with 43x fewer think tokens.
  • Grammar is ~20 lines of GOAL/APPROACH/EDGE structure; code output stays unconstrained.
  • Zero training required, works with any llama.cpp, vLLM, SGLang server that accepts grammars.
  • Limitations: contamination risk, tuned for coding tasks, untested on math and agents.

Reasoning models have a token-bloat problem. Qwen3, DeepSeek-R1, QwQ and their peers routinely burn thousands of tokens inside <think> blocks, exploring alternatives, restating the problem, and hedging before they commit to an answer. A new open-source experiment called structured-cot shows that most of that prose is scaffolding, not computation, and that you can strip it out with a small grammar file at inference time.

The finding in numbers

The repo tests a simple idea: force the model's reasoning block to obey a tiny grammar, while leaving the actual answer or code unconstrained. It offers grammar-constrained chain-of-thought for reasoning LMs with zero training, delivering 22x compression on HumanEval+ and +14 percentage points on a recent LiveCodeBench public-test slice.

On HumanEval+ with unsloth/Qwen3.6-35B-A3B-GGUF at Q4_K_M, free-form thinking scored 92.1% pass@1 with a mean of 3087 thinking tokens per problem. The grammar-constrained run scored 92.7% with just 138 thinking tokens on average, and 408 total tokens versus 3410. Same accuracy band, roughly an order of magnitude less compute per problem.

The LiveCodeBench v6 slice (50 LeetCode problems with contest dates from 2025 onward) is where things get more interesting. Free-form thinking hit 50% pass@1 while averaging 11553 thinking tokens; a richer plan grammar reached 64% pass@1 with 267 thinking tokens and cut total tokens from 13632 to 2743. It also eliminated the 18 empty-code extraction failures that free-form produced.