VFM-Recon Turns Any RGB Video Into Accurate 3D Meshes Across Environments
A new framework fixes the scale-ambiguity problem that has kept vision foundation models out of scene-level 3D reconstruction, delivering a 70.1 F1 on outdoor scenes.
Takeaways
- VFM-Recon injects frozen VGGT features into scene-level neural volumetric reconstruction from monocular video.
- A submap-based scale-alignment stage uses SuperPoint plus LightGlue triangulation and a log-space factor graph.
- Only a small MLP adapter inside VGGT's frame-attention blocks is trained; the foundation model stays frozen.
- Achieves F1 74.3 on ScanNet, 46.7 on TUM RGB-D, and 70.1 on Tanks and Temples versus 51.8 for VGGT.
- Ablations confirm scale alignment is the critical piece: removing it drops outdoor F1 from 70.1 to 45.4.
- Trained only on ScanNet using two A6000 GPUs over roughly one week; no code release announced.
Reconstructing a full 3D scene from a plain RGB video is one of those problems that quietly refuses to generalize. Models trained on indoor room scans tend to fall apart the moment you point them at a courtyard or a building facade. A new paper called VFM-Recon takes a run at that gap by borrowing the geometric priors baked into vision foundation models like VGGT, then fixing the one thing that has kept those priors from being useful for volumetric fusion: scale.
Why foundation model depth doesn't just plug in
Scene-level neural reconstruction methods regress a signed distance field (SDF) directly from posed images, then extract a mesh with marching cubes. The catch is that these networks are tightly coupled to their training distribution, and vision foundation models, while excellent at cross-domain generalization, produce depth predictions that are only defined up to an unknown scale. Volumetric fusion cannot tolerate that. Even small per-frame scale mismatches accumulate as depth maps are integrated into a single voxel grid, producing warped surfaces and floaters.
The authors frame this as a structural incompatibility. VFMs like VGGT, MoGe-2, and the Depth Anything family give you strong relative structure and robust features, but no metric guarantees. Classical volumetric fusion needs consistent metric depth across the whole trajectory. VFM-Recon is an attempt to reconcile those two worlds without retraining the foundation model.

A two-stage decoupled pipeline
The system splits the problem into a scale-alignment stage followed by a VFM-augmented reconstruction stage. The pipeline works roughly like this: