Qwen's Qwen3-1.7B-Base Beats Older 3B Models With 36T Tokens
Alibaba's compact 1.7B pretraining checkpoint packs 36 trillion training tokens, 32K context, and 119 languages into an Apache 2.0 base model built for fine-tuning.
Takeaways
- Qwen3-1.7B-Base is the raw pretraining checkpoint of Qwen's 1.7B edge model, released under Apache 2.0.
- Dense transformer with 28 layers, GQA (16 Q / 8 KV heads), and a native 32K context window.
- Pretrained on 36 trillion tokens spanning 119 languages via a three-stage curriculum.
- Qwen claims density parity with Qwen2.5-3B-Base, per the Qwen3 blog.
- Outperforms larger Qwen2.5-3B base on STEM and coding benchmarks per the technical report.
- Requires transformers 4.51.0+ and works out of the box with vLLM, SGLang, and Ollama quantizations.
The Qwen team has quietly become one of the most reliable suppliers of small open-weight base models, and Qwen3-1.7B-Base sits at the small end of that lineup. It is the raw pretraining checkpoint of the Qwen3 series, before any instruction tuning, and it has already pulled more than 1.2 million downloads on Hugging Face along with hundreds of community fine-tunes.
Base means no chat template, no RLHF, no alignment. This is the next-token predictor you fork when you want to build your own instruct model, domain expert, or research probe without inheriting anyone else's post-training decisions.
What is actually in the box
The architecture is a straightforward dense transformer sized for edge and consumer GPUs. From the model card:
- 1.7B total parameters, 1.4B non-embedding
- 28 layers
- Grouped-query attention with 16 query heads and 8 key/value heads
- 32,768 token context length
- BF16 weights, Apache 2.0 license
The model targets diverse hardware, including mobile and edge platforms, and its native 32K context can be stretched further using YaRN-based rotary embedding scaling for longer documents and multi-turn interactions. The combination of GQA with a small KV head count keeps the memory footprint manageable during long-context inference on a single consumer card.