Adelaide Proves a Two-Line Fix Shrinks Neural Field Training by Orders of Magnitude
A CVPR paper proves scaling laws for neural fields and proposes a new weight initialization that cuts parameter counts from super-quadratic to quadratic growth.
Takeaways
- New CVPR paper proves scaling laws for neural fields with sine, sinc, Gaussian, and wavelet activations.
- Standard inits need super-quadratic parameters in depth; the paper's new scheme cuts that to quadratic.
- Trick: sample the final layer from a Normal with variance 2/n^{3/2} instead of 1/n.
- Improves PSNR by 2 to 4 dB on occupancy reconstruction and 1.1 dB on NeRF Lego.
- Validated on image regression, 4x super-resolution, occupancy fields, NeRF, and Navier-Stokes PINNs.
- Read the full paper on arXiv.
Neural fields, the coordinate-based MLPs behind NeRF, implicit shape reconstruction, and physics-informed networks, have largely been designed by trial and error. A new CVPR paper from the University of Adelaide and Amazon puts a theoretical floor under that practice, proving how many parameters a neural field actually needs to converge under gradient descent, and showing that a small change to how the last layer is initialized dramatically shrinks that number.
The overparameterization tax nobody was measuring
A neural field is a small network that maps input coordinates (like pixel positions or 3D points) to values (like color or density), effectively representing a signal as the network's weights. Practitioners often scale networks ad hoc with larger datasets, lacking a clear understanding of how such architectures should proportionately adapt to data size. The core question the authors tackle: given N training samples, how wide does the network need to be so that gradient descent is guaranteed to reach a global minimum?
Previous theory had answered this only for ReLU networks. Nguyen's earlier state-of-the-art result showed that deep ReLU networks need one hidden layer of order N cubed, while shallow ones need N squared. That is a lot of neurons, and it did not describe the activations that people actually use in neural fields, such as sine (SIREN), sinc, Gaussian, or Gabor wavelet (WIRE).