Maryland Researchers Pinpoint Why AI Training Silently Explodes After Weeks
A new study pins long-horizon pretraining failures on BF16 quantization error in the LM head, and fixes it by swapping the optimizer.
Takeaways
- New paper identifies time-bomb loss spikes as delayed pretraining failures driven by LM head quantization error.
- Reducing precision accelerates the failure, FP32 LM head computation prevents it entirely in tested horizons.
- Proposed fix: swap AdamW for SGD with momentum on the LM head only, keeping AdamW elsewhere.
- SGD-M keeps logit norm and BF16 vs FP32 logit gap nearly flat across training.
- Existing mitigations like z-loss and μ-centering all work by suppressing the same quantization error.
- Experiments span 28M to 404M parameter models trained up to 28x Chinchilla-optimal token counts.
Long pretraining runs can maintain a stable loss curve for weeks before diverging without recovery. A new paper from researchers at the University of Maryland and collaborators isolates one version of this failure, which the authors call a time-bomb spike, and traces it to growing logits combined with finite-precision arithmetic in the language-model head.
The operational risk comes from timing. These spikes emerge after prolonged training, when early-run checks have passed and much of the compute budget has already been spent. The paper reports that lower numerical precision brings the failure forward, while higher precision and targeted changes to the output head delay or prevent it over the tested training horizons.
The head drifts until BF16 loses detail
A language-model head projects each transformer hidden state into one score, or logit, for every token in the vocabulary. Softmax converts those logits into probabilities, and cross-entropy uses the probability assigned to the target token to calculate the training loss.
The proposed mechanism begins with AdamW gradually increasing the magnitude of the head weights and logits. BF16 has a wide numerical range but limited precision within that range, so the spacing between representable values grows as values become larger. Rounding error in the output projection therefore increases alongside the logits.
The experiments separate logit growth from the eventual failure. Logit norms rose in AdamW runs across precision settings, including FP32, but the FP32 runs remained stable over the reported horizon. In BF16, the growing quantization error eventually changed relative token scores enough to distort the softmax distribution and drive irreversible loss divergence.
Precision starts the failure clock
The researchers tested 28 million-parameter models for substantially more tokens than the Chinchilla compute-optimal budget, the approximate token count expected to make efficient use of a model of that size. Extending training beyond that budget made the delayed failures reproducible within an academic compute budget.