Skip to main content
Back

Unsloth's Qwen3-8B Runs on a 16GB GPU Without the Usual Quality Hit

Unsloth's dynamic 4-bit quantization of Qwen3-8B shrinks the reasoning model to fit consumer GPUs while preserving accuracy on math and code.

Model, Full Access
  • Unsloth released a dynamic 4-bit quantized Qwen3-8B, already at 898k downloads.
  • Uses selective per-layer quantization, keeping sensitive layers in higher precision.
  • Fits fine-tuning on 16GB GPUs with 4x lower memory and 8x longer context than FA2.
  • Qwen3-8B supports hybrid thinking and non-thinking modes toggled by /think and /no_think.
  • Native 32K context, extendable to 128K via YaRN scaling, Apache 2.0 licensed for commercial use.
  • Loads directly via Unsloth FastModel or Unsloth Studio for local inference.

Unsloth has published a dynamically quantized 4-bit version of Alibaba's Qwen3-8B, a hybrid reasoning model that can toggle between chain-of-thought and direct-answer modes. The repo has crossed 898k downloads, a signal of how many developers want to run or fine-tune a modern reasoning-capable model on a single consumer GPU without the accuracy cliff that usually comes with aggressive quantization.

Inside the repo

The base model is Qwen3-8B, a causal LM with 8.2 billion parameters that supports two modes in a single set of weights: thinking for complex reasoning, math, and code, and non-thinking for general dialogue. Architecturally it uses 36 layers with grouped-query attention (32 query heads, 8 KV heads), handles up to 32,768 tokens natively, and scales to 131,072 with YaRN. It ships under Apache 2.0, so commercial use is on the table.

On top of that base, Unsloth applies its Dynamic 4-bit Quantization over bitsandbytes. Standard 4-bit quantization compresses every layer uniformly, which tanks accuracy on models with sensitive activation patterns. The dynamic variant selectively avoids quantizing certain parameters, recovering most of the accuracy while using under 10% more VRAM than plain BnB 4-bit.