A 7.4B Model Matches GPT-3 13B Using 20x Less Compute
New research shows architectural tricks like recursive depth and model growth can bend pretraining scaling exponents, delivering compute savings that compound.
Takeaways
- New paper argues architectural choices can change pretraining scaling exponents, not just constants.
- A 7.4B model growth architecture matches GPT-3 13B on CORE using 20x less compute.
- Reported compute multipliers grow with scale: 1.55x at 1e20 FLOPs, projected 2.7x at 1e25.
- Three interventions studied: looping, model growth (tied and untied), and a boundary operator that normalizes residuals.
- In data-constrained multi-epoch training, looping acts as a regularizer worth about 2.2x compute efficiency.
- Code released at qlabs-eng/scaling-exponents with growth scripts and boundary operator.
Empirical scaling laws usually describe how language-model loss falls as training compute, parameter count, and data increase. Chinchilla-style laws also estimate the compute-optimal balance between model size and training tokens. A paper by Zixi Chen, Akshay Vegesna, Samip Dahal, and Andrew Gordon Wilson reports that three architectural changes alter the fitted scaling exponent itself: recursive looping, model growth, and a boundary operator.
The reported gains widen with scale. An untied-growth model with 7.4 billion parameters matched GPT-3 13B on the CORE evaluation suite while using about one-twentieth of the pretraining compute. The result applies to that benchmark and compute accounting, but it challenges the common assumption that architecture can deliver only fixed, constant-factor savings.
A steeper exponent compounds
Compute scaling laws often take the form L(C) = B + AC−α, where L is loss, C is training compute, B is an estimated irreducible floor, and α controls the curve’s slope. Lowering A shifts the curve by a fixed amount. Increasing α makes loss fall faster, so the equivalent-compute advantage grows as training runs become larger.
The authors measure that advantage as a compute multiplier: the amount of compute a standard transformer would need to reach the same loss. Their best measured variant reaches about 1.55× at 1020 FLOPs, with the fitted law projecting 2.7× at 1025 FLOPs.
Depth that still changes predictions
The proposed mechanism centers on computational depth, defined as the number of layers that meaningfully change the model’s prediction. Pre-normalized transformers can lose useful depth because the residual stream grows while each later block contributes a progressively smaller update. The network remains deep in layer count, yet its upper layers exert less influence on the output.