Skip to main content
Back

Alibaba's Qwen3.5-35B-A3B Matches Frontier AI Running on 3B Active Parameters

Alibaba's new sparse MoE vision-language model activates only 3B of 35B parameters, delivers Qwen3.5-27B-level performance at a fraction of the compute.

Model, Full Access
  • Qwen released Qwen3.5-35B-A3B-FP8, a 35B MoE vision-language model with only 3B active params.
  • Hybrid architecture: Gated DeltaNet linear attention plus GQA plus 256-expert MoE across 40 layers.
  • Native 262K context, extensible to 1M tokens via YaRN, with vision and video input.
  • Fine-grained FP8 block quantization keeps quality nearly identical to BF16 at half the memory.
  • Matches dense Qwen3.5-27B on MMLU-Pro (85.3) and beats GPT-5-mini on TAU2-Bench (81.2).
  • Apache 2.0 open weights; hosted as Qwen3.5-Flash at roughly $0.14/$1.00 per M tokens.

Alibaba's Qwen team has released an FP8-quantized version of Qwen3.5-35B-A3B, a native vision-language mixture-of-experts model that activates only 3 billion of its 35 billion parameters per token. The mid-size entry in the Qwen3.5 family ships under Apache 2.0 and pushes hard on every axis that matters for self-hosted inference: sparsity, context length, multimodality, and quantization.

Overall quality lands close to the dense Qwen3.5-27B according to reported evaluations, while the compute budget per forward pass sits closer to a 3B model. That trade lives or dies on the architecture underneath, and this one is unusual.

Inside the stack

The model is a causal LM with a vision encoder, 40 layers, and a hidden dimension of 2048. The layer stack is where things get interesting: 10 repeats of (3 × Gated DeltaNet + MoE) followed by 1 × (Gated Attention + MoE). Most of the layers use linear attention, and only every fourth block uses standard attention.

  • Gated DeltaNet handles most of the sequence mixing with linear attention, which is much cheaper than standard self-attention at long context. It uses 32 value heads and 16 QK heads with a head dim of 128.
  • Gated Attention layers use grouped-query attention with 16 Q heads and only 2 KV heads, which cuts KV-cache memory pressure sharply during inference.
  • MoE uses 256 experts with 8 routed plus 1 shared expert active per token, at an expert intermediate dimension of 512.
  • Multi-Token Prediction (MTP) is trained in, so speculative decoding via NEXTN works out of the box in SGLang and vLLM.