UC San Diego Shrinks Hybrid AI Models 7x Using a Smarter Compression Trick
A forward-only sensitivity method uses KL divergence to pick which layers of hybrid SSM-Transformer models can safely drop to INT4 on edge silicon.
Takeaways
- KL divergence provides a formal exponential upper bound on perplexity change from quantization, unlike SQNR
- Forward-only sensitivity method needs no gradients or retraining, works with proprietary calibration data
- Student-to-teacher KL hits 0.79 Kendall tau vs perplexity across six SSM and hybrid models, beating SQNR at p less than 10^-6
- In Hymba, block 31 alone drives over 70 percent of INT4 quantization error, with mamba.x_proj the dominant subtype
- Mamba-1.4B compresses 7.2x to 1.4 GB on Intel Lunar Lake CPU with no perplexity loss and 178 ms latency
- Code and OpenVINO pipeline available at github.com/jasonkongie/kl-ssm-quant
Shrinking hybrid Mamba-style language models down to something an Intel Lunar Lake laptop chip can run is mostly a game of picking the right layers to crush. A new paper argues that most teams have been picking with the wrong ruler, and shows that a single-pass metric borrowed from information theory can do the job without gradients, retraining, or access to private training data.
The work, A KL Lens on Quantization, comes out of UC San Diego and San Diego State, with code released at jasonkongie/kl-ssm-quant. It targets a specific pain point in edge deployment: hybrid architectures like Hymba and Zamba that interleave state space models with transformer blocks, where different components tolerate quantization very differently.
Why uniform quantization keeps breaking on SSMs
4-bit uniform quantization of Transformers has become standard in industry for inference speed-ups, but the story falls apart on hybrids. SSMs behave differently under quantization than attention blocks: the state transition in selective SSMs has highly sensitive internal activations, and its outputs contain large magnitude outliers absent from Transformer activations. Push everything to INT4 uniformly and perplexity can explode by orders of magnitude on a single bad layer.
Mixed precision solves this by keeping sensitive layers at higher bit-width, but the classic way to find those layers, Hessian-based scoring like HAWQ, requires backpropagation. That is expensive at billion-parameter scale and awkward when the calibration data is proprietary. The other common proxy, signal-to-quantization-noise ratio (SQNR), works for CNNs, yet the authors show it is fundamentally mismatched to language modeling.