A Curvature-Aware Trick Fixes Physics AI Without Full FP64 Costs
A new precision controller for physics-informed neural networks dynamically switches between FP32 and FP64 using curvature signals, matching full double-precision accuracy at lower compute cost.
Takeaways
- A new curvature-aware precision controller dynamically switches PINN training between FP32 and FP64, cutting compute cost without sacrificing accuracy.
- PINN failure modes are caused by FP32 rounding errors that prematurely halt L-BFGS optimization, not by bad architectures or loss landscape traps.
- The controller reuses curvature signals already computed by L-BFGS to decide when FP64 is needed, adding zero overhead from new computations.
- Tested on four canonical failure-mode benchmarks (Convection, Reaction, Wave, Allen-Cahn) and multiple architectures including MLP, PINNsFormer, PINNMamba, and KAN.
- Dynamic precision matches or slightly exceeds full FP64 accuracy while reducing training wall-clock time across all benchmarks.
- No code or model weights have been released publicly alongside the paper.
Physics-informed neural networks (PINNs) sit at an awkward intersection: they need the numerical rigor of scientific computing, but they run on deep learning infrastructure built for speed. That tension has quietly been one of the biggest blockers to making PINNs reliable in practice. A new paper proposes a surprisingly elegant fix: stop treating numerical precision as a fixed setting, and instead let the optimizer itself decide when to upgrade.
The work, titled "Curvature-aware dynamic precision approach for physics-informed neural networks," introduces a precision controller that piggybacks on information the L-BFGS optimizer already computes, using it to flip between 32-bit and 64-bit floating point on the fly during training.
Why precision is a bigger deal for PINNs than for regular networks
Recent work established something counterintuitive: many well-known PINN failure modes are not caused by bad architectures or poor loss landscapes. PINNs often exhibit failure modes where the PDE residual loss converges while the solution error stays large, a phenomenon traditionally blamed on local optima. The real culprit turns out to be insufficient arithmetic precision: with standard FP32, the L-BFGS optimizer prematurely satisfies its convergence test, freezing the network in a spurious failure phase.
Arithmetic precision is generally not considered a decisive factor in vision and language networks, where FP16 and lower is common practice. But the problem PINNs are dealing with is scientific computing with high precision requirements. Second-order and hybrid optimizers demonstrate significantly improved performance under double precision, because they rely on precise curvature information such as Hessian approximations and gradient norms during line search or trust-region updates. Second-order methods typically terminate when the norm of the gradient falls below a threshold, so the level of numerical precision directly impacts their stopping criteria and convergence quality.
The fix seems obvious: just use FP64 everywhere. But double precision is more energy-intensive and currently restricted to high-end hardware such as NVIDIA's H100 GPUs. Increased demand for FP64 compute could raise the carbon footprint of scientific machine learning and exacerbate resource inequity between well-funded labs and those with limited budgets. The new paper's bet is that you don't need FP64 all the time, only when it matters.