NYU's PrimeSVT Shrinks Vision AI by 27% Without Specialized Hardware
Researchers from NYU Abu Dhabi's eBRAIN Lab introduce PrimeSVT, an automated structured pruning framework that cuts Spiking Vision Transformer memory by 26.68% while keeping accuracy within 3% of the original model.
Takeaways
- PrimeSVT is a new automated framework from eBRAIN Lab (NYU Abu Dhabi) for compressing Spiking Vision Transformers via structured pruning.
- It achieves 26.68% memory reduction on the SDTv2 model while keeping accuracy within 3% of the unpruned baseline (72.9% with fine-tuning vs. 73.3%).
- Unlike prior work, it uses structured pruning (whole channels removed), making the compressed model compatible with standard GPUs and TPUs without custom accelerators.
- A novel robustness-based prioritized compression policy assigns higher pruning rates to layers that tolerate compression well and protects fragile layers.
- Beyond memory, PrimeSVT also reduces FLOPs by 15.5% and energy consumption by 13.1% compared to the baseline model.
- The framework is the first to offer automated single-shot structured pruning for SViTs; paper available at arxiv.org/abs/2606.03428.
Spiking Vision Transformers (SViTs) are a compelling class of models that combine the architectural strengths of Vision Transformers with the ultra-low-power computation of Spiking Neural Networks (SNNs). The catch: they are still too large to run efficiently on embedded hardware. A new research paper from the eBRAIN Lab at NYU Abu Dhabi introduces PrimeSVT, an automated framework that compresses pre-trained SViTs using structured pruning, making them viable for deployment on standard computing hardware without specialized accelerators.
The Problem With Pruning SViTs Today
To understand why this matters, it helps to know what makes SViTs hard to compress. Standard pruning methods for regular neural networks cannot be directly applied to SNNs because SNNs represent data as binary spikes over time rather than continuous activations. The two architectures are fundamentally different at the neuron level.
Existing SViT compression work has relied almost exclusively on unstructured pruning, which zeroes out individual weights scattered irregularly across the model. The problem is that this kind of sparsity only pays off if you have specialized hardware (like a custom accelerator) that can skip over those zeros. On widely-used systolic array engines, the kind found in GPUs and TPUs, unstructured pruning typically results in irregularly distributed zero weights, and the systolic array compute engine expects dense weight mapping and neural operations, meaning you get no actual speedup.
On top of that, the manual approach of existing methods requires a huge design time to find an appropriate pruning setting for each network, making this approach not scalable. Every new SViT model requires a human expert to hand-tune the pruning configuration from scratch.