QuBLAST Shrinks 8B Models by 45% Without Retraining or Quality Loss
QuBLAST assigns different bit-widths to each transformer block using cross-entropy sensitivity, shrinking 8B models by up to 45% with under 5% quality loss
Takeaways
- QuBLAST is a new PTQ framework that assigns different bit-widths to each attention block individually, rather than quantizing all blocks uniformly.
- It uses cross-entropy loss sensitivity analysis per block to decide which blocks can tolerate aggressive compression and which need higher precision.
- An activation scaling map per block tames outlier activations cheaply, without the matrix rotations used by heavier methods like QuaRot or SpinQuant.
- Results show 40-45.2% model size reduction across Qwen3-8B, Llama3-8B, Mistral v0.1-7B, and Falcon H1R-7B, with under 5% perplexity increase.
- Notably tested on Falcon-H1's hybrid SSM-Transformer architecture, a non-standard design that existing quantization benchmarks typically ignore.
- No code release yet; evaluation is limited to WikiText perplexity and does not cover downstream task benchmarks like reasoning or coding.
QuBLAST is a new post-training quantization (PTQ) framework that treats each attention block in a large language model as its own compression target, rather than applying the same bit-width uniformly across the entire network. The result: model sizes shrink by 40 to 45 percent with perplexity degradation kept under 5 percent, and it works on both classic transformer architectures and newer hybrid designs that mix attention with state-space models.
The paper arrives at a moment when the field is saturated with PTQ methods, but most of them share a quiet assumption that has gone largely unchallenged: every block in the network deserves the same quantization treatment. QuBLAST challenges that directly.
The problem no one is solving cleanly
State-of-the-art methods typically employ uniform post-training quantization across attention blocks of the network, overlooking the potential of applying different quantization levels in the same network. That uniformity is a blunt instrument. Some blocks are highly sensitive to precision loss; others can be pushed to 4-bit or lower with almost no measurable impact. Treating them identically wastes the compression budget on blocks that don't need it, and damages blocks that do.
The second problem is activation outliers. This difficulty arises due to the higher prevalence of outliers in activations, making them harder to quantify than weights. When a small number of activation values are dramatically larger than the rest, any fixed quantization grid gets stretched to accommodate them, and the resolution for normal values collapses. Existing fixes, like rotation-based methods (QuaRot, SpinQuant), work but add significant computational overhead.
Existing methods have not considered evaluation using emerging LLMs with non-conventional attention architectures, such as state-space models, which pose different challenges in applying quantization. This is a real gap: hybrid models like Falcon-H1 are gaining traction, and their mixed SSM-attention blocks behave very differently under quantization than a standard transformer layer.