Stanford's SSD Teaches AI New Facts Using 69% Fewer Training Tokens
Stanford and ETH Zürich researchers introduce a per-token mixed-policy self-distillation method that cuts supervised tokens by 38 to 69 percent while preserving general skills.
Takeaways
- Speculative Self-Distillation lets student write rollouts, teacher takes over only at tokens where document changes the prediction.
- Cuts supervised tokens 38-69% versus on-policy distillation while matching accuracy on three knowledge tasks.
- Single JSD threshold τ interpolates between off-policy (τ=0) and on-policy (τ≥ln 2), no schedule needed.
- Teacher contribution naturally decays from 43% of tokens early to 1-5% by end of training.
- Forgetting stress test: SSD loses 1.4 general-ability points vs 4.5 for on-policy, 6.6 for off-policy.
- Code and modified vLLM engine open-sourced at GitHub; project page at ssd-distill.github.io.
Researchers at Stanford and ETH Zürich have introduced Speculative Self-Distillation (SSD), a fine-tuning method that decides at each token whether a document-aware teacher or a document-blind student should continue a training rollout. Across three knowledge-learning tasks, SSD preserved the accuracy of on-policy distillation while reducing supervised-token use by 38% to 69%.
Developers can place new information in a prompt or retrieve it at inference time, but context windows remain finite, retrieval adds latency, and models can fail to use relevant passages. Fine-tuning stores frequently needed knowledge in the model’s weights. SSD targets the computational cost of doing that with self-distillation, as outlined on the project page.
Why on-policy rollouts burn compute
Self-distillation uses two copies of a model. The teacher receives the source document and produces a next-token distribution informed by that document. The student receives no document and learns to match the teacher’s distribution.
| Approach | Who generates the rollout | Tradeoff |
|---|---|---|
| Off-policy distillation | Teacher | Generates data quickly, though the student trains on polished teacher prefixes that it may not produce during inference. |
| On-policy distillation, including SDFT | Student | Matches inference-time behavior more closely, while requiring the teacher to score many low-value tokens. |
In the paper’s Wikipedia analysis, most of the per-token training signal appeared within roughly the first 15 tokens of an on-policy rollout and then dropped sharply. Once the student moved onto a prefix unrelated to the document, the document-conditioned teacher produced a distribution close to the student’s. Those later tokens offered little useful correction.