TUM's DeepBA Rivals Billion-Parameter Models With Just 2 Million Parameters
TUM's DeepBA turns bundle adjustment into a 2M-parameter graph network. It recovers cameras, focal lengths and depth from 2D matches alone, starting from scratch, and gets close to billion-parameter models.
Takeaways
- TUM's DeepBA replaces classical bundle adjustment with a learned 2.04M-parameter graph neural network.
- It recovers poses, focal lengths and depth from 2D matches only, starting from identity cameras and unit depth.
- On CO3Dv2 it hits 97.4 RRA@15, beating COLMAP and GASfM and trailing ~1.2B-parameter VGGT by a few points.
- Gets faster than classical SfM as view count grows. Works with any matcher and uses no tracks, thanks to scale-invariant training.
- Weak on ETH3D, planar scenes (focal-depth ambiguity), tight thresholds and non-pinhole cameras.
- Localizes new images without PnP or intrinsics. Code is listed as "coming soon" on the project page.
Bundle adjustment closes nearly every 3D reconstruction pipeline by refining camera poses and 3D structure until projected points align with image observations. Researchers at the Technical University of Munich have replaced its numerical update rule with DeepBA, a 2.04-million-parameter graph neural network that estimates camera poses, focal lengths, and scene geometry from 2D point matches across a variable number of views.
DeepBA requires no scene-specific geometric initialization. Its demonstrations begin with identity camera poses and unit depths, then apply the same learned update 100 times. On several benchmarks, the resulting pose estimates approach those of VGGT, a roughly 1.2-billion-parameter transformer. DeepBA’s input consists solely of point correspondences between image pairs, allowing the matcher and solver to remain separate pipeline components.
Why Initialization Controls Classical BA
Structure-from-motion, or SfM, reconstructs camera positions and scene geometry from overlapping photographs. Systems such as COLMAP and GLOMAP typically detect and match features, connect observations into tracks representing shared 3D points, estimate initial camera poses, and run bundle adjustment.
Bundle adjustment minimizes reprojection error, the pixel distance between an observed feature and the projected position of its estimated 3D point. Solvers such as Gauss-Newton and Levenberg-Marquardt linearize those residuals around the current estimate, use the Jacobian to calculate an update, and repeat. Because the linearization is local, poor initial poses or incorrect matches can lead to weak solutions.
Modern SfM pipelines spend substantial effort on initialization, outlier rejection, track construction, and dataset-specific heuristics. Image-to-geometry models such as DUSt3R, MASt3R, and VGGT avoid much of that pipeline by predicting geometry from pixels, but they combine feature extraction, matching, and reconstruction in models containing hundreds of millions or billions of parameters.
A Learned Update, Repeated 100 Times
DeepBA preserves the iterative structure of bundle adjustment while learning the update function. Training amortizes the cost of solving many reconstruction problems into one graph network, which repeatedly reads the current geometric residuals and predicts corrections. The architecture mirrors the underlying view graph: cameras form nodes, matched image pairs form edges, and individual correspondences carry measurement features.
Inside One Refinement Step
- The encoder reads the current state, including one pose, one focal length, and one depth map per image, together with the view graph and pairwise matches.