Illustrative image: the transformer architecture — Deletion Tuning learns a binary mask over the gate projection inside each block's MLP. Image: Wikimedia Commons / Yuening Jia, CC BY-SA 3.0.
TL;DR: Deletion Tuning LLM fine tuning explained in one line: researchers found that zeroing fewer than 0.05% of a language model's weights — deleting them, not updating them — can match full fine-tuning across reinforcement learning, supervised fine-tuning and distillation, while keeping old skills more intact and shrinking the saved adapter to a tiny binary mask.
Key Highlights
- A NeurIPS 2026 paper posted on arXiv on 7 November 2026 asks how little of a pretrained LLM must change to learn new skills — and answers: under 0.05% of its weights, set to zero.
- Deletion Tuning learns a binary mask over the gate projection of transformer blocks. Kept weights stay exactly at their pretrained values; the mask is the entire adapter.
- It matched full fine-tuning at the largest scale tested in each of three post-training regimes, on eight backbones from 0.5B to 30B parameters, including a 30B mixture-of-experts model.
- The trained mask is hundreds to thousands of times smaller than a full fine-tuning checkpoint — smaller than even a LoRA rank-1 adapter in the paper's main RL comparison.
- It forgets far less: under reinforcement learning, prior-task accuracy stayed near the base model while full fine-tuning, LoRA and random sparse fine-tuning lost up to 5.4 points.
What happened?
On 7 November 2026, researchers from New York University and the University of Calgary posted a paper titled "The Surprising Effectiveness of Deleting Weights in LLM Post-Training" on arXiv. The authors — Jack Lu, Zhenbang Yang, Mike Lasby, Tejas Pote, Yani Ioannou and Mengye Ren — set out to test an idea that sounds almost backwards: instead of teaching a model by changing its weights, teach it by choosing which weights to delete.
The paper reached a wider audience on 10 November 2026, when it surfaced on Hacker News and was picked up by AI research digests, because it challenges the default playbook for adapting large language models. Today the choice is usually dense full fine-tuning for quality or LoRA for efficiency. Deletion Tuning offers a third option: extreme sparsity through learned deletion masks. Like Meta FAIR's e-means algorithm, which clusters data 10x faster, it is a NeurIPS 2026 methods paper — a new algorithm, not a product launch — so this post explains the method, the evidence and the limits, rather than reporting a release.
What is Deletion Tuning?
Fine-tuning a language model normally means updating its weights — the billions of numbers that encode what the model knows. Full fine-tuning updates all of them. LoRA freezes the model and trains a small pair of low-rank matrices that nudge its behaviour. Sparse fine-tuning updates only a chosen subset of weights. Deletion Tuning does something different from all three: it writes no new values at all.
The method learns a binary mask — a yes/no decision for each targeted weight. At deployment, a weight either keeps its pretrained value or is set to exactly zero. The trained mask is the whole adapter: one bit per weight, instead of a multi-gigabyte file of updated numbers. The authors' central question was whether erasure alone could serve as the entire fine-tuning mechanism. Across their experiments, the answer was yes.
This is not the same as pruning a finished model to make inference cheaper, which is what methods like Wanda and SparseGPT do. Pruning preserves capabilities a model already has. Deletion Tuning uses deletion during post-training to teach the model new ones — reasoning, domain knowledge and tool use, in the paper's tests.
How the method works
Deletion Tuning targets one weight matrix per transformer block: the gate projection of the MLP. In modern models such as Llama and Qwen2.5, the MLP uses a SwiGLU design, where the gate projection multiplicatively controls which intermediate channels flow through to the next layer. Zeroing a gate weight therefore switches off a channel — and the paper's bet is that switching off the right channels can steer the model toward the new skill, because the useful behaviour is already latent in the pretrained weights.
Training works like this. Each targeted weight is paired with a trainable score, and during training the weight is multiplied by a smooth mask value between 0 and 1. As training progresses, that mask is gradually sharpened until it becomes a hard decision: keep or delete. At deployment, every weight is either bit-identical to pretraining or zero. The mask is initialised using the Wanda pruning method's importance scores, but the authors checked that the gains come from learning the mask, not from that starting point.
The intuition behind why it works comes from recent findings the paper builds on: fine-tuned solutions sit close to the pretrained model, and reinforcement-learning fine-tuning naturally changes only a small subnetwork of parameters. Deletion Tuning pushes that observation to its limit — if only a small part needs to change, perhaps nothing needs to change value at all, and choosing what to remove is enough.
Illustrative image: a simplified neural network — every connection is a weight, and Deletion Tuning teaches new skills by zeroing a tiny fraction of them. Image: Wikimedia Commons / Mikael Häggström, CC0.
Deletion Tuning LLM fine tuning explained: the three tests
The authors tested the method in the three post-training regimes that labs actually run after pretraining, using Group Relative Policy Optimization (GRPO) for reinforcement learning with verifiable rewards, standard supervised fine-tuning, and self-distillation fine-tuning for on-policy distillation. The experiments spanned two model families and eight backbones from 0.5B to 30B parameters. At the largest model scale tested in each setting, Deletion Tuning matched full fine-tuning.
| Regime | What the model learns | Deletion Tuning result |
|---|---|---|
| Reinforcement learning with verifiable rewards (RLVR / GRPO) | Reasoning, rewarded by checkable answers such as maths unit tests | Matched full fine-tuning; also solved problems the base model failed in all 128 sampled attempts |
| Supervised fine-tuning (SFT) | Imitating curated dialogues and labelled examples | Matched full fine-tuning at the largest scale tested |
| On-policy distillation (SDFT) | A student model matching a teacher's behaviour on its own rollouts | Matched full fine-tuning at the largest scale tested |
Two properties matter beyond the headline match. First, the adapter is tiny: the trained mask is hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoint, and smaller than even a LoRA rank-1 adapter in the paper's main RL comparison. Shipping a bitmask instead of a multi-gigabyte weight delta changes the economics of storing and distributing many adapted models. Second, the method forgets less. Under reinforcement learning, its prior-task accuracy stayed near the base level while full fine-tuning, LoRA and random sparse fine-tuning lost up to 5.4 points on earlier tasks — the classic catastrophic forgetting problem that bites teams adding skills to the same base model month after month.
The authors also analysed why masking alone can work. Comparing the geometry of the learned weight changes — stable rank, spectrum drift and principal-weight overlap — Deletion Tuning landed closest to full fine-tuning, ahead of LoRA and random sparse fine-tuning. In plain terms, deleting the right weights moves the model in roughly the same direction in weight space as updating all of them. That is the paper's explanation for why a method that writes no new values can still reach full fine-tuning quality.
Deletion Tuning vs LoRA vs full fine-tuning
For teams choosing how to adapt a model, the three methods now sit on different trade-offs. Full fine-tuning remains the quality reference but is expensive to store and the most prone to forgetting. LoRA is the mature, widely supported middle ground. Deletion Tuning is the new research-stage option: the smallest artifact and the least forgetting in the paper's tests, but with the least production tooling.
| Approach | What changes | Saved artifact | Forgetting risk | Maturity |
|---|---|---|---|---|
| Full fine-tuning | Every weight updated | Huge — full model copy | Higher — up to 5.4 points lost in the paper's RL tests | Production default |
| LoRA | Small low-rank adapter matrices | Small | Medium | Ubiquitous, well supported |
| Deletion Tuning | Nothing updated — selected weights zeroed | Tiny binary mask, hundreds to thousands of times smaller than full fine-tuning | Lower in the paper — prior-task accuracy near base level | Research-stage, code not yet public |
That comparison comes with a warning the authors would agree with: do not rip out LoRA pipelines overnight. Deletion Tuning adds mask-learning overhead, and it needs replication on each team's own reward model and data before any production claim. The reinforcement-learning results are the strongest part of the paper — GRPO is the recipe behind today's reasoning models, the same family of training Grok 4.8's reinforcement-learning training relies on — but one paper, however striking, is a signal to test, not a migration plan.
Why it matters for readers in India
The paper contains no India-specific results, so the relevance here is practical, not claimed by the authors. India has a large community of teams fine-tuning open models — the Llama and Qwen families the paper tests are among the most widely used bases — often on tighter GPU and storage budgets than frontier labs. Three consequences follow if the result holds up.
First, cost. Storing one full fine-tuned copy per customer, language or use case is expensive; a mask that is hundreds to thousands of times smaller makes per-customer or per-language adapters far cheaper to keep, including for Indian-language adaptations where a team might maintain many variants of one base model. Second, continual learning. Teams that add a skill every month — say, a new domain or a new Indian language — hit forgetting first; a method that keeps prior-task accuracy near the base level directly addresses that. Third, distribution. Tiny adapters are easier to ship to edge devices and on-premise deployments, the kind of setups Indian enterprises and startups commonly run. None of this is proven in production yet — the serving question is open — but the direction of travel favours smaller, cheaper adaptation, which is where most Indian AI work lives.
Should you care?
Verdict: ✅ Worth knowing, not yet worth switching to. If you fine-tune open models, Deletion Tuning is one of the most surprising methods papers of the year: deleting under 0.05% of weights matched full fine-tuning in three regimes, with a far smaller artifact and less forgetting. But it is a single preprint, the code is not public yet, and serving binary masks in production inference stacks is unproven. Read the paper, watch for the code release, and pilot it against your LoRA baseline before changing anything.
What is still unproven
Honest caveats matter as much as the headline. The paper optimises training science, not inference: it is not yet clear whether serving stacks can apply binary masks on gate projections at runtime without slowing things down, or whether masks must be merged back into the base weights first — which would erase the storage win at serving time. The edits are gate-only, so tasks that need attention-layer changes may not be covered. The 0.05% figure is an order-of-magnitude headline; the sparsity a given task needs may be denser, and it varies by layer and training phase. And as a preprint, the NeurIPS camera-ready version may differ from the arXiv v1 this post is based on — the code release the authors' project page points to is the thing to watch. Like TUTTI's sub-3ms encrypted computing breakthrough, this is a research result to track, not a shipped product to buy.
FAQs
What is Deletion Tuning?
Deletion Tuning is a method for adapting a pretrained language model by learning a binary mask over the gate projection of its transformer blocks. Selected weights are set to exactly zero and all other weights keep their pretrained values — no new values are written. The authors report it matches full fine-tuning while changing fewer than 0.05% of parameters.
How does Deletion Tuning work?
Each targeted weight is paired with a trainable score, and during training the weight is multiplied by a smooth mask value between 0 and 1. The mask is gradually sharpened until it becomes a hard keep-or-delete decision. At deployment the adapter is one bit per weight, which is why the saved checkpoint is hundreds to thousands of times smaller than a full fine-tuning checkpoint.
Is Deletion Tuning better than LoRA?
In the paper's tests, Deletion Tuning matched full fine-tuning quality like LoRA aims to, but with a smaller artifact — smaller than even a LoRA rank-1 adapter in the main RL comparison — and less forgetting of earlier tasks. LoRA remains the mature, widely supported choice; Deletion Tuning is research-stage, its code is not yet public, and it needs replication on your own data before production use.
Can deleting weights really teach an LLM new skills?
According to the paper, yes — in the tested regimes. The authors argue the useful behaviour is already latent in the pretrained weights, and deleting the right gate weights steers the model toward it. Under reinforcement learning with verifiable rewards, the method even solved problems the base model failed in all 128 sampled attempts. The result is a preprint finding, not yet independently replicated, so treat it as a strong signal rather than settled fact.
Takeaway
Deletion Tuning turns a basic assumption of model training on its head: learning a new skill may not require writing new values into a model at all. Zeroing fewer than 0.05% of its weights matched full fine-tuning across three regimes, produced an adapter hundreds to thousands of times smaller, and forgot less. If independent teams replicate it once the code lands, the cheapest way to adapt a language model may soon be to delete — not to add.
If you fine-tune models yourself, the useful next step is to read the paper on arXiv and watch the authors' project page for the code release — then test a mask against your own LoRA baseline on the same evaluation harness before deciding anything.
Comments
Post a Comment