Skip to main content
Back

NYU's Deletion Tuning Matches Full Fine-Tuning by Deleting 0.05% of Weights

Deletion Tuning learns a binary keep-or-delete mask over gate projections, matching full fine-tuning accuracy while zeroing under 0.05% of weights.

  • Deletion Tuning matches full fine-tuning by zeroing under 0.05% of weights with no other changes (paper).
  • Trained adapter is one bit per weight: a 30 MB mask for a 61 GB, 30B MoE model.
  • Prior-task accuracy stays near base while FFT, LoRA, and random sparse tuning drop up to 5.4 points.
  • Works across RL with verifiable rewards, supervised fine-tuning, and on-policy distillation.
  • Masks only the gate projection of transformer blocks; router untouched in MoE models.
  • Code, trained masks, and project page available at agenticlearning.ai/deleting-weights.

Researchers at NYU’s Agentic Learning AI Lab and the University of Calgary report that pretrained language models can acquire task-specific behavior by disabling a small set of existing weights. Deletion Tuning freezes the base checkpoint and learns a binary mask over the gate projections in transformer blocks.

The authors report that zeroing fewer than 0.05% of a model’s weights reaches accuracy comparable to full fine-tuning on the largest models they evaluated. Their experiments cover reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation across dense Llama and Qwen models and a 30B mixture-of-experts model.

Hundreds of variants, one frozen checkpoint

Common post-training methods create additional parameters for every specialized model. Full fine-tuning, or FFT, updates many continuous weight values and typically produces another full checkpoint. LoRA stores smaller low-rank matrices, but each adapter still contains newly learned values and can alter capabilities acquired during pretraining.

Method Learned change Deployment artifact Main constraint
Full fine-tuning Continuous updates across tuned parameters Full checkpoint or dense delta High storage cost per variant
LoRA Continuous low-rank matrices Adapter or merged checkpoint Update rank is capped
Random sparse tuning Continuous updates at preselected positions Sparse delta Positions are fixed before training
Deletion Tuning Binary keep-or-delete decisions Bit mask Target weights retain their original value or become zero

Compact, non-destructive updates matter when one base model supports many customers, tasks, or sequential training stages. For the 30B mixture-of-experts example, the authors report a 30 MB mask associated with a 61 GB base checkpoint. Each deployment can share the frozen checkpoint and load its own mask.

How a gate becomes a bit mask

A transformer’s feed-forward network expands each token representation, applies nonlinear processing, and projects it back to the model dimension. Many current architectures use a gated feed-forward network in which a gate projection controls how strongly intermediate features pass through the block.