Brown and NYU Researchers Map 3D Vision's Chaotic Evolution Into One Unified Guide
A CVPR 2026 workshop paper maps every major 3D data format, dataset, and learning paradigm into one unified reference for practitioners navigating a fragmented field.
Takeaways
- What it is: A CVPR 2026 workshop survey that builds a unified taxonomy of 3D vision, connecting representations, datasets, learning paradigms, and applications in one map.
- Eight representations covered: Point clouds, meshes, voxels, RGB-D, CAD/B-Rep, implicit neural fields, 3D Gaussians, and multi-view images -- each analyzed for efficiency-fidelity trade-offs.
- Three learning paradigms: Differentiable rendering (NeRF/3DGS), end-to-end geometric foundation models (DUSt3R, VGGT), and generative priors / native 3D latents (DreamFusion, TRELLIS).
- 50 datasets surveyed: Mesh (56%) and multi-view (50%) dominate; implicit field datasets are nearly absent (2%), a gap the paper flags as significant.
- Living GitHub repo: awesome-3d-datasets is an actively maintained companion with full dataset tables and method links.
- Scope spans reconstruction to 4D world models: Covers static reconstruction, text-to-3D generation, 3D-consistent video, dynamic 4D Gaussians, and Vision-Language-Action robotics.
3D vision is one of the most exciting and chaotic corners of modern AI. In the span of a few years, the field has gone from classical point cloud pipelines to Neural Radiance Fields (NeRF), then to 3D Gaussian Splatting, and now to 4D dynamic world models -- each wave introducing new representations, new benchmarks, and new terminology. The problem is that these advances have accumulated in silos, making it genuinely hard to see how everything connects. A new paper from researchers at Brown, UMD, Penn, USC, NYU, Sydney, and Stability AI, presented at the CVPR 2026 OpenSUN3D Workshop, tries to fix that.
"A Cookbook of 3D Vision: Data, Learning Paradigms, and Application" is a structured survey that builds a single conceptual map connecting geometric representations, datasets, learning frameworks, and downstream applications. The framing is deliberately data-centric: rather than organizing the field by model architecture, it starts from how 3D data is represented and acquired, then traces how those choices cascade into what you can learn and what you can build.
Why the field needed a map
3D vision has rapidly evolved, driven by increasingly diverse data representations, learning paradigms, and modeling strategies -- yet the field remains fragmented across representations and benchmarks, making it difficult to develop unified perspectives on efficiency, fidelity, and scalability. That fragmentation is the core problem the paper addresses. A researcher working on autonomous driving with LiDAR point clouds and a researcher doing text-to-3D generation with Gaussian splatting are both doing "3D vision," but they often read different papers, use different datasets, and speak different technical dialects.
Datasets and benchmarks have not only enabled fair evaluation but also actively shaped the evolution of 3D learning paradigms by defining data structures, supervision formats, and scalability constraints. This is a key insight in the paper: the data you have determines the methods you can build, and the benchmarks you design determine what counts as progress. Understanding that feedback loop is essential for anyone trying to navigate the field strategically.
The eight representations, decoded
The paper offers a high-level map of how 3D data are represented, stored, and processed, covering major formats such as point clouds, meshes, voxel grids, RGB-D images, CAD models, implicit fields, and 3D Gaussians within one unified view. Each format comes with a distinct efficiency-fidelity trade-off that shapes which tasks it is suited for.