Skip to main content
Back

Alibaba's Qwen3.5-2B Base Packs 262K Context Into a Single GPU

Alibaba's compact 2B pretrained checkpoint packs native vision-language, a 262K context, and a hybrid Gated DeltaNet architecture into an Apache 2.0 release.

Model, Full Access
  • Alibaba released Qwen3.5-2B-Base, the pretrained checkpoint of its compact multimodal model under Apache 2.0.
  • Hybrid architecture mixes Gated DeltaNet linear attention with grouped-query attention in a 3-to-1 ratio.
  • Native 262,144 token context and support for 201 languages, with early-fusion vision-language pretraining.
  • Instruct variant in thinking mode hits MMLU-Pro 74.0% and GPQA Diamond 65.8%.
  • Runs in Transformers, vLLM, SGLang, and Ollama; needs about 19GB VRAM at full context.
  • Intended for fine-tuning, LoRA adapters, and research, not direct chat deployment.

Alibaba's Qwen team just dropped the base checkpoint for one of the smallest members of its Qwen3.5 family, and it is a lot more interesting than the parameter count suggests. Qwen3.5-2B-Base is the raw pretrained model behind the instruction-tuned Qwen3.5-2B, released as an Apache 2.0 artifact designed for fine-tuning, adapter training, and in-context learning research rather than direct chat use.

It sits inside the recently expanded small tier of the Qwen3.5 lineup. Alibaba positioned the small series as tiny, fast options for edge devices at 0.8B and 2B, a lightweight agent base at 4B, and a 9B that is closing the gap with much larger models. The base weights ship alongside the instruct variants so researchers and industrial users can build on the same foundation.

Small model, big-model plumbing

The headline architectural choice is that this is a hybrid rather than a plain transformer. Qwen3.5-2B combines Gated Delta Networks and Gated Attention in a 6x(3x DeltaNet then FFN then 1x Attention then FFN) pattern. Gated DeltaNet is a linear-attention style block, so roughly three quarters of the token-mixing layers run in linear time while the remaining quarter uses standard grouped-query attention to preserve exact recall where it matters.

Specs of the dense stack worth knowing before you plan a fine-tune:

  • 2.27B parameters, 24 layers, hidden size 2,048, SwiGLU FFN at 6,144 intermediate width
  • Grouped-query attention with 8 query heads and 2 key-value heads, head dim 256
  • RoPE with theta set to 10,000,000, no sliding window
  • Vocabulary of 248,320 tokens, RMSNorm throughout
  • One multi-token prediction head used during pretraining

The unusually large vocabulary is why the model card explicitly calls out embedding handling. Control tokens like the im_start and im_end markers were trained to allow efficient LoRA-style PEFT with the official chat template, which removes the need to finetune embeddings, a significant optimization given the larger vocabulary. In practice you can attach LoRA adapters and adopt the chat format without paying the memory cost of training the full embedding matrix.