Alibaba's Qwen3.5-122B Beats GPT-5-mini on Agents at Half the Memory
Alibaba's mid-tier MoE ships in FP8 with 262K context, unified vision-language, and benchmarks that beat GPT-5-mini across reasoning, coding, and agentic tasks.
Takeaways
- Qwen released official FP8 weights for its 122B/10B-active MoE, Apache 2.0 licensed.
- Beats GPT-5-mini on MMLU-Pro (86.7), GPQA Diamond (86.6), and Terminal Bench 2 (49.4).
- Hybrid Gated DeltaNet plus full attention in a 3:1 ratio, 256 experts, 262K context extensible to 1M.
- Native vision-language and video: 86.2 on MathVision, 83.9 on VideoMME without subtitles.
- Runs on 8x H100 or 4x B200 via vLLM, SGLang, or KTransformers with built-in tool and MTP support.
- Part of a family that also ships 27B dense, 35B-A3B, and a 397B flagship.
Alibaba's Qwen team has released an official FP8-quantized version of Qwen3.5-122B-A10B, the mid-tier member of the Qwen3.5 family. The FP8 build cuts the memory footprint roughly in half while, per the model card, performing nearly identically to the original weights. It landed on Hugging Face after users flooded the discussion tab on the base repo asking for it, and has already crossed 1.8 million downloads.
Quantization is only part of the story. The underlying Qwen3.5-122B-A10B is a sparsely activated Mixture-of-Experts model that punches well above its active-parameter weight, and shipping it in FP8 under Apache 2.0 puts frontier-adjacent multimodal reasoning within reach of a single 8-GPU node.
What is actually in the box
The model is a hybrid MoE with a vision encoder attached to a causal language model backbone. Headline numbers: 122 billion total parameters and 10 billion active per token, with 256 experts total (8 routed plus 1 shared active per forward pass). Context length is 262K native, extensible to 1M via YaRN rope scaling.
The layer stack is unusual. Rather than pure transformer blocks, Qwen3.5 interleaves linear attention with full attention in a repeating pattern: three Gated DeltaNet plus MoE blocks followed by one Gated Attention plus MoE block, repeated twelve times for 48 layers total. Linear attention scales with sequence length instead of its square, which is what makes the 262K context tractable at inference time.
The FP8 recipe itself is worth noting. The model card describes it as fine-grained FP8 quantization with a block size of 128, and the tensor dump on Hugging Face shows a mix of BF16, F8_E4M3, and F32 tensors. Non-expert paths and sensitive layers stay in higher precision while the bulk of the routed expert weights drop to 8-bit.
Benchmarks that punch up
Qwen positions this model against GPT-5-mini and GPT-OSS-120B on language, and against Claude Sonnet 4.5 and Qwen3-VL-235B-A22B on vision. It wins on most rows despite having a fraction of GPT-5-mini's presumed compute budget.
| Benchmark | Qwen3.5-122B-A10B |
|---|