Skip to main content
Back

Unsloth Shrinks DeepSeek-R1 by 80% so You Can Run It at Home

Unsloth's dynamic quantization shrinks DeepSeek-R1 from 720GB to 131GB with an 80% size cut while keeping the 671B reasoning model coherent.

Model, Full Access
  • Unsloth released dynamic GGUF quants of DeepSeek-R1, shrinking the 671B model from 720GB to 131GB.
  • Selective per-layer quantization keeps attention, first 3 layers and down_proj at 4-6 bit while MoE experts drop to ~1.5 bit.
  • Four variants ship at 1.58, 1.73, 2.22 and 2.51 bits, ranging from 131GB to 212GB on disk.
  • 1.58-bit fits in 160GB VRAM (2x H100) and hits ~140 tokens/sec throughput; 20GB RAM also works, slowly.
  • Compatible with llama.cpp, Ollama, LM Studio, vLLM and SGLang under an MIT license.
  • Technique generalized into Unsloth Dynamic 2.0, now applied to non-MoE models too.

Running a 671B parameter reasoning model on your own hardware used to be a fantasy reserved for anyone with a small cluster of H100s. Unsloth's DeepSeek-R1-GGUF release makes it uncomfortably close to reality by shipping selectively quantized versions of the full R1 model that fit into a fraction of the original memory, without collapsing into gibberish the way naive low-bit quantization does.

The headline number is an 80% size reduction: the 671B parameter model shrinks from 720GB to 131GB in the most aggressive 1.58-bit version. Unsloth also uploaded 1.73-bit, 2.22-bit, and 2.51-bit variants for anyone willing to trade disk space for higher fidelity.

Why naive quantization breaks R1

Compressing every weight in a Mixture-of-Experts model down to about 1.5 bits is a fast way to produce garbage. Uniform low-bit quantization sends R1 into endless loops and nonsense outputs. Unsloth's dynamic quants sidestep this by treating layers differently based on how much they contribute to model behavior.

Unsloth studied R1's architecture and left the layers that matter most at higher precision, while pushing the bulk of the MoE experts down to 1.5-bit. The first three layers, accounting for 0.5% of total weights, stay at 4 or 6 bits. The many experts inside the MoE layers, roughly 1.5% of total weights, are held at 6 bits. Attention modules run at 4 or 6 bits because they disproportionately affect output quality. The down_proj layer, which handles important intermediate computation, gets relatively high precision.

The naming convention is worth internalizing before you download 130+ GB of shards. Even the smallest quant is a range, not a single bit-width: