Skip to main content
Back

Why Teacher Forcing Still Beats Decoder-Aware Training, Proven Mathematically

A Google DeepMind team proves that two natural ways to train language models on their own decoded outputs are broken by construction, and concludes that teacher forcing remains the right tool for supervised training.

Paper, Full Access
  • Blondel and coauthors prove that two natural sequence-level losses for language models are fundamentally flawed.
  • The greedy-decoding loss is discontinuous, nonconvex and unbounded below. Optimizers can lower it forever without real progress.
  • The sampling-based loss is smooth but still unbounded. Via Gumbel tricks, it equals the greedy loss averaged over random noise.
  • Stopping gradients through sampling gives a moment matching direction that provably lowers the standard likelihood objective.
  • That direction has high variance, while teacher forcing gives the exact gradient in one forward pass.
  • Theory-only paper with no code or models. The authors suggest RL post-training for sequence-level signals (thread).

Most autoregressive language models learn through teacher forcing: the model receives the correct prefix, predicts the next token and updates its weights using cross-entropy. A new theory paper from Mathieu Blondel, Michaël E. Sander, Germain Vivier-Ardisson, Tianlin Liu and Vincent Roulet examines whether supervised training should instead use prefixes produced by the model itself. Their analysis finds that common decoder-aware losses are mathematically ill-behaved, while a principled alternative leads back to the likelihood objective that teacher forcing already optimizes.

Correct prefixes give way to generated ones

Autoregressive negative log-likelihood decomposes into token-level cross-entropies. At each position, the model conditions on the correct preceding tokens. This factorization makes the objective efficient because each conditional distribution needs normalization only over the vocabulary, rather than over every possible sequence.

During generation, the model conditions on its previous outputs. An incorrect token can therefore change every later prediction, creating the train-inference mismatch commonly called exposure bias.

# Training: condition on the correct prefix
for t in range(T):
    loss += cross_entropy(model(x, y_true[:t]), y_true[t])

# Greedy generation: condition on the generated prefix
y_hat = []
for t in range(T):
    y_hat.append(model(x, y_hat).argmax())

Concern about compounding errors has motivated scheduled sampling, imitation learning and other methods that expose models to generated prefixes during training. In his announcement thread, Blondel describes the paper’s target as sequence-level objectives built around the same decoding procedures used at inference time.

Decoder-aware losses have no floor

The paper studies objectives based on two widely used decoding methods. Greedy decoding selects the highest-probability token at every step, while autoregressive sampling draws each token from the model’s predicted distribution.