Alibaba's QwQ-32B Beats o1-mini With One-Twentieth the Parameters
Alibaba's Qwen team ships a 32B open-weight reasoning model that trades blows with DeepSeek-R1 at a fraction of the compute cost.
Takeaways
- Qwen team released QwQ-32B, a 32.5B open-weight reasoning model under Apache 2.0.
- Matches DeepSeek-R1 (671B) on math and coding, beats o1-mini across key benchmarks.
- Trained with two-stage RL using accuracy verifiers and code execution instead of learned reward models.
- 131,072 token context via YaRN, with agent and tool-use capabilities baked in.
- Runs on consumer-grade hardware when quantized; priced around $0.15 per million tokens via providers.
- Full technical details in the Qwen blog post.
Alibaba's Qwen team has released QwQ-32B, a reasoning-focused large language model that punches well above its weight class. On math and coding benchmarks, it nearly matches DeepSeek-R1-671B while beating o1-mini and distilled versions of R1. It does this with roughly one-twentieth the parameters of R1, and it ships under Apache 2.0.
The weights are available now on Hugging Face and ModelScope, through the hosted Qwen Chat interface, and via Alibaba Cloud's DashScope API. Third-party providers list it at roughly $0.15 per million input tokens, making it one of the cheaper reasoning-tier options on the market.
A small model behaving like a big one
QwQ-32B is a dense 32.5B parameter causal language model built on Qwen2.5-32B. The comparison target matters here: DeepSeek-R1 has 671 billion parameters, though as a mixture-of-experts model only a subset activates at any given time, roughly 37B per forward pass. QwQ-32B is dense, so all 32.5B fire on every token, but the raw memory footprint is dramatically smaller. That makes it deployable on consumer-grade hardware.
The Qwen team frames this as evidence for a specific hypothesis: reinforcement learning on top of a strong pretrained base can substitute for a lot of parameter count when the task is reasoning. On the Artificial Analysis Intelligence Index, QwQ-32B scores 20, above average among open weight models of similar size, which have a median score of 15.
Under the hood
The architecture is standard modern Qwen2 with a few notable specs:
- 32.5B total parameters, 31.0B non-embedding
- 64 transformer layers
- Grouped-query attention with 40 query heads and 8 key/value heads
- RoPE positional encoding, SwiGLU activations, RMSNorm, QKV bias in attention
- Full context length of 131,072 tokens
The long context comes with a caveat. For prompts longer than 8,192 tokens, you have to explicitly enable YaRN (a technique for stretching RoPE embeddings past their trained range) by editing