Skip to main content
Back

Stanford's SSD Teaches AI New Facts Using 69% Fewer Training Tokens

Stanford and ETH Zürich researchers introduce a per-token mixed-policy self-distillation method that cuts supervised tokens by 38 to 69 percent while preserving general skills.

  • Speculative Self-Distillation lets student write rollouts, teacher takes over only at tokens where document changes the prediction.
  • Cuts supervised tokens 38-69% versus on-policy distillation while matching accuracy on three knowledge tasks.
  • Single JSD threshold τ interpolates between off-policy (τ=0) and on-policy (τ≥ln 2), no schedule needed.
  • Teacher contribution naturally decays from 43% of tokens early to 1-5% by end of training.
  • Forgetting stress test: SSD loses 1.4 general-ability points vs 4.5 for on-policy, 6.6 for off-policy.
  • Code and modified vLLM engine open-sourced at GitHub; project page at ssd-distill.github.io.

Researchers at Stanford and ETH Zürich have introduced Speculative Self-Distillation (SSD), a fine-tuning method that decides at each token whether a document-aware teacher or a document-blind student should continue a training rollout. Across three knowledge-learning tasks, SSD preserved the accuracy of on-policy distillation while reducing supervised-token use by 38% to 69%.

Developers can place new information in a prompt or retrieve it at inference time, but context windows remain finite, retrieval adds latency, and models can fail to use relevant passages. Fine-tuning stores frequently needed knowledge in the model’s weights. SSD targets the computational cost of doing that with self-distillation, as outlined on the project page.

Why on-policy rollouts burn compute

Self-distillation uses two copies of a model. The teacher receives the source document and produces a next-token distribution informed by that document. The student receives no document and learns to match the teacher’s distribution.

Approach Who generates the rollout Tradeoff
Off-policy distillation Teacher Generates data quickly, though the student trains on polished teacher prefixes that it may not produce during inference.
On-policy distillation, including SDFT Student Matches inference-time behavior more closely, while requiring the teacher to score many low-value tokens.

In the paper’s Wikipedia analysis, most of the per-token training signal appeared within roughly the first 15 tokens of an on-policy rollout and then dropped sharply. Once the student moved onto a prefix unrelated to the document, the document-conditioned teacher produced a distribution close to the student’s. Those later tokens offered little useful correction.