Skip to main content
Back

Adelaide Proves a Two-Line Fix Shrinks Neural Field Training by Orders of Magnitude

A CVPR paper proves scaling laws for neural fields and proposes a new weight initialization that cuts parameter counts from super-quadratic to quadratic growth.

  • New CVPR paper proves scaling laws for neural fields with sine, sinc, Gaussian, and wavelet activations.
  • Standard inits need super-quadratic parameters in depth; the paper's new scheme cuts that to quadratic.
  • Trick: sample the final layer from a Normal with variance 2/n^{3/2} instead of 1/n.
  • Improves PSNR by 2 to 4 dB on occupancy reconstruction and 1.1 dB on NeRF Lego.
  • Validated on image regression, 4x super-resolution, occupancy fields, NeRF, and Navier-Stokes PINNs.
  • Read the full paper on arXiv.

Neural fields, the coordinate-based MLPs behind NeRF, implicit shape reconstruction, and physics-informed networks, have largely been designed by trial and error. A new CVPR paper from the University of Adelaide and Amazon puts a theoretical floor under that practice, proving how many parameters a neural field actually needs to converge under gradient descent, and showing that a small change to how the last layer is initialized dramatically shrinks that number.

The overparameterization tax nobody was measuring

A neural field is a small network that maps input coordinates (like pixel positions or 3D points) to values (like color or density), effectively representing a signal as the network's weights. Practitioners often scale networks ad hoc with larger datasets, lacking a clear understanding of how such architectures should proportionately adapt to data size. The core question the authors tackle: given N training samples, how wide does the network need to be so that gradient descent is guaranteed to reach a global minimum?

Previous theory had answered this only for ReLU networks. Nguyen's earlier state-of-the-art result showed that deep ReLU networks need one hidden layer of order N cubed, while shallow ones need N squared. That is a lot of neurons, and it did not describe the activations that people actually use in neural fields, such as sine (SIREN), sinc, Gaussian, or Gabor wavelet (WIRE).