Wisconsin Researchers Build a Scaling Law That Makes Synthetic Rewrites Worth the Compute
A new scaling law predicts the optimal mix of synthetic rewrites in pretraining, and shows a 22B model can outperform its own rewriter at a tenth the compute.
Takeaways
- New scaling law predicts loss from model size, tokens, rewrite fraction, and epochs while charging rewriter compute. Paper
- Optimal rewrite share grows from under 10% at 3×10¹⁸ FLOPs to 32% at 10²² FLOPs.
- A held-out 22B run lands within 1.1% of the forecast, extrapolating 33× in FLOPs.
- Target model beats its SmolLM2-Instruct rewriter in BPB at one-tenth the rewriter's training compute.
- Self-rewriting with same-size rewriter beats natural-only epoching up to 3B parameters.
- Rewrites help Math and Code at all budgets, but QA only above a compute threshold.
Language-model developers increasingly train on both human-authored web documents and model-generated rewrites of those documents. Researchers from Wisconsin, Stanford, and collaborating institutions have proposed a rewrite scaling law that estimates the best mixture while counting the compute required to generate synthetic text.
This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute. Bits-per-byte normalizes prediction loss by the raw size of the text, and lower values indicate better language modeling. The experiments also show how the optimal rewrite share changes as compute grows.
Why spend FLOPs on rewritten text?
Synthetic rewriting sends a human-authored document, such as a forum post or Wikipedia passage, through a language model that converts it into another form. A prompt might request cleaner tutorial prose, a question-and-answer structure, or a more explicit explanation. These transformations can make the same underlying information easier to learn, but every generated token consumes inference compute that could have funded additional target-model training.
A law with generation costs built in
The proposed scaling law predicts target-model loss from four variables:
- Target-model parameter count
- Number of target-model training tokens
- Share of training tokens drawn from rewrites
- Number of passes, or epochs, over the natural documents
For each candidate mixture, the total budget includes both target-model training and rewrite generation. This accounting closes a gap in analyses that treat synthetic corpora as precomputed inputs. FLOPs, or floating-point operations, provide the common unit for comparing those costs.