TRSP Shrinks LLaMA2 by 25% With No Retraining and 1.35x Speedup
TRSP prunes transformer layers by first learning per-layer weights, then regularizing pruned layers to shift their knowledge into layers that survive.
Takeaways
- TRSP introduces two-stage regularization for layer-wise pruning of LLMs without any retraining. Paper
- Stage one learns per-layer scalar weights via L1 regularization; stage two pushes pruned layers toward identity mappings.
- On LLaMA2-7B at 25% sparsity, perplexity is 20% lower than ShortGPT and accuracy drops just 1%.
- Pruning OPT-13B by 50% yields 75% higher throughput and 46% lower latency versus dense.
- Iterative greedy weight learning beats one-shot because one-shot selects harmful consecutive layer runs.
- Code is released at github.com/fmk345/TRSP; validated on models up to Llama-2-70B.
Shrinking large language models by chopping out whole transformer layers is appealing because it delivers real wall-clock speedups on standard hardware. The catch is that even the layers you deem unimportant usually still carry useful knowledge, so removing them tends to break the model until you retrain it. A new method called TRSP (Two-Stage Regularization-Based Structured Pruning) tries a different route. Instead of slicing first and repairing later, it uses regularization to push knowledge out of the doomed layers before removing them, and skips retraining entirely.
The reported numbers are strong. On LLaMA2-7B at 25% sparsity, TRSP cuts perplexity by 20% compared to ShortGPT, average zero-shot accuracy drops by just 1%, and the pruned model runs 1.35 times faster than the dense baseline. Code is on GitHub.
Why layer pruning keeps failing
Structured pruning splits into two flavors. Channel-wise pruning removes rows or columns from weight matrices; layer-wise pruning removes entire transformer blocks. Treating whole blocks as the pruning unit gives a simpler workflow than operating at the channel level, but the tradeoff is that a layer is a much bigger chunk of the model, so any mistake in what you cut hurts more.
The dominant pipeline in prior work looks the same across methods: score each layer with some importance metric, delete the low scorers, then fine-tune to recover. Even low-scoring layers can hold valuable knowledge, and pruning them directly discards it outright, forcing substantial retraining to claw it back. Retraining is expensive, and for very large models it's often the whole reason you wanted to prune in the first place.