vLLM's LLM Compressor Shrinks 1.6TB Models by 70% in Two Hours
The vLLM project ships llm-compressor, a Transformers-native toolkit that turns Hugging Face models into vLLM-ready quantized checkpoints with one API call.
Takeaways
- The vLLM project's llm-compressor unifies quantization workflows for vLLM deployment
- Supports GPTQ, AWQ, SmoothQuant, AutoRound, SpinQuant, and REAP expert pruning in one API
- Covers FP8, INT8, INT4, NVFP4, MXFP4, mixed precision, plus KV cache and attention quantization
- Outputs
compressed-tensorsformat that loads directly into vLLM without conversion - DDP and disk offloading enable compressing trillion-parameter MoE models on modest hardware
- Installable via
pip install llmcompressor, Apache-2.0 licensed, works natively with Hugging Face
Serving a 30B parameter model in full precision is painful. Serving a 1T parameter Mixture-of-Experts model is nearly impossible without heavy quantization. The vLLM project has been steadily building llm-compressor to solve exactly that, and the library has grown into a one-stop shop for producing quantized checkpoints that plug directly into vLLM.
LLM Compressor is a Transformers-compatible Python library that applies compression algorithms to large language models and saves them in the compressed-tensors format that vLLM loads natively. Before this project existed, creating quantized checkpoints meant navigating a fragmented ecosystem of bespoke libraries like AutoGPTQ, AutoAWQ, and AutoFP8. LLM Compressor was built from the ground up to unify compression best practices such as GPTQ, SmoothQuant, SparseGPT, and RTN, working directly with Hugging Face models so vLLM can load the resulting checkpoints without a conversion step.
One recipe, many precisions
Coverage has expanded significantly. The library supports AWQ, GPTQ, AutoRound, and Round-to-Nearest algorithms, along with QuIP and SpinQuant-style rotation transforms plus KV cache and attention quantization. Precision options span FP8, INT8, INT4, NVFP4, MXFP4, and mixed-precision schemes, all through a one-shot workflow that needs only a small calibration set.