Alibaba's Qwen3.5-35B-A3B Matches Frontier AI Running on 3B Active Parameters
Alibaba's new sparse MoE vision-language model activates only 3B of 35B parameters, delivers Qwen3.5-27B-level performance at a fraction of the compute.
Takeaways
- Qwen released Qwen3.5-35B-A3B-FP8, a 35B MoE vision-language model with only 3B active params.
- Hybrid architecture: Gated DeltaNet linear attention plus GQA plus 256-expert MoE across 40 layers.
- Native 262K context, extensible to 1M tokens via YaRN, with vision and video input.
- Fine-grained FP8 block quantization keeps quality nearly identical to BF16 at half the memory.
- Matches dense Qwen3.5-27B on MMLU-Pro (85.3) and beats GPT-5-mini on TAU2-Bench (81.2).
- Apache 2.0 open weights; hosted as Qwen3.5-Flash at roughly $0.14/$1.00 per M tokens.
Alibaba's Qwen team has released an FP8-quantized version of Qwen3.5-35B-A3B, a native vision-language mixture-of-experts model that activates only 3 billion of its 35 billion parameters per token. The mid-size entry in the Qwen3.5 family ships under Apache 2.0 and pushes hard on every axis that matters for self-hosted inference: sparsity, context length, multimodality, and quantization.
Overall quality lands close to the dense Qwen3.5-27B according to reported evaluations, while the compute budget per forward pass sits closer to a 3B model. That trade lives or dies on the architecture underneath, and this one is unusual.
Inside the stack
The model is a causal LM with a vision encoder, 40 layers, and a hidden dimension of 2048. The layer stack is where things get interesting: 10 repeats of (3 × Gated DeltaNet + MoE) followed by 1 × (Gated Attention + MoE). Most of the layers use linear attention, and only every fourth block uses standard attention.
- Gated DeltaNet handles most of the sequence mixing with linear attention, which is much cheaper than standard self-attention at long context. It uses 32 value heads and 16 QK heads with a head dim of 128.
- Gated Attention layers use grouped-query attention with 16 Q heads and only 2 KV heads, which cuts KV-cache memory pressure sharply during inference.
- MoE uses 256 experts with 8 routed plus 1 shared expert active per token, at an expert intermediate dimension of 512.
- Multi-Token Prediction (MTP) is trained in, so speculative decoding via NEXTN works out of the box in SGLang and vLLM.