SGS-SLAM Bakes Semantic Labels Directly Into 3D Gaussian Maps
A trending open-source SLAM system fuses 3D Gaussian Splatting with semantic segmentation to deliver object-aware dense reconstruction in real time.
Takeaways
- SGS-SLAM is the first semantic visual SLAM built on 3D Gaussian Splatting, released under BSD-3 on GitHub.
- Each Gaussian jointly stores appearance, geometry and a semantic class, optimized through a multi-channel loss.
- A semantic feature loss and semantic-guided keyframe selection fix oversmoothing and drift in object-level geometry.
- Reports state-of-the-art pose, reconstruction and segmentation on Replica, ScanNet and ScanNet++ with real-time rendering.
- Published at ECCV 2024; arXiv paper already has 95+ citations and multiple follow-ups.
- RGB-D only, requires ground-truth semantic masks, and targets bounded indoor scenes.
Mingrui Li, Shuhong Liu and collaborators have published the complete SGS-SLAM repository, including tracking, mapping, post-optimization and visualization code under the BSD-3-Clause license. Presented at ECCV 2024, the system is described by its authors as the first semantic visual SLAM pipeline built on 3D Gaussian Splatting.
Visual simultaneous localization and mapping estimates a camera’s trajectory while reconstructing its surroundings. SGS-SLAM represents that reconstruction as a collection of Gaussian primitives, each defined by properties such as position, shape, opacity, color and semantic features. Those explicit primitives can be rendered quickly and updated individually as new RGB-D frames arrive.
Semantics become map data
Dense SLAM systems based on neural implicit fields, including NICE-SLAM, ESLAM and Point-SLAM, can blur object boundaries and lose thin or weakly textured geometry. SGS-SLAM jointly optimizes appearance, geometry and semantic features within the Gaussian map, preserving class information in the same representation used for reconstruction and rendering.
- Joint channel optimization. Each Gaussian carries appearance, geometric and semantic parameters that are updated during tracking and mapping.
- Semantic feature loss. Semantic supervision supplies an additional optimization signal where color and depth provide limited evidence. The paper reports better reconstruction of thin and low-texture objects as a result.
- Semantic keyframe selection. The system uses semantic consistency when choosing keyframes, reducing reconstruction errors associated with accumulated camera drift.
The implementation builds on the SplaTAM tracker and uses Open3D for interactive visualization. Developers familiar with Gaussian Splatting pipelines will encounter a conventional split between tracking, mapping, global optimization and viewing.
Benchmarks stay indoors
The published evaluation covers bounded indoor scenes captured with RGB-D sensors. The repository provides configurations for three datasets, with each benchmark using a specific sequence and semantic-label protocol.
| Dataset | Release coverage | Semantic setup |
|---|---|---|
| Replica | Supported indoor scenes | Ground-truth semantic masks |
| ScanNet | scene0000_00, scene0059_00, scene0106_00, scene0181_00 and scene0207_00 |
Semantic extraction scripts included |
| ScanNet++ | Two scenes under the SplaTAM protocol | Protocol-specific preprocessing |
Across these benchmarks, the paper reports leading results against its selected baselines for camera pose estimation, 3D reconstruction, semantic segmentation and object-level geometric accuracy. Comparisons depend on the listed sequences, supervision and evaluation protocol. The real-time claim applies to rendering; tracking, mapping and global post-optimization remain separate workloads that require application-specific benchmarking.
Files, scripts and license
- RGB-D tracking and Gaussian map optimization
- A global post-SLAM optimization pass
- Replica, ScanNet and ScanNet++ configurations
- ScanNet semantic preprocessing scripts
- An Open3D-based interactive reconstruction viewer
- Gaussian scene export as
.plyfiles - Optional Weights & Biases experiment logging
The BSD-3-Clause license permits broad reuse of the code, subject to its attribution and redistribution terms. Dataset assets, pretrained components and third-party dependencies retain their own licenses.
From RGB-D frames to a viewer
The documented environment uses Conda, CUDA 11.8 and PyTorch 2.0.1 on a CUDA-capable NVIDIA GPU. After placing the Replica data where the configuration expects it, the primary workflow consists of three commands:
python scripts/slam.py configs/replica/slam.py
python scripts/post_slam_opt.py configs/replica/post_slam_opt.py
python viz_scripts/tk_recon.py configs/replica/slam.py
- Track and map:
slam.pyestimates camera poses while constructing the Gaussian scene. - Refine globally:
post_slam_opt.pyoptimizes the completed reconstruction. - Inspect the result:
tk_recon.pyopens the interactive viewer.
Weights & Biases logging remains optional through a configuration flag, allowing local runs without an external experiment-tracking service.
RGB-D sets the boundary
- Depth input is required. The supplied configurations cover RGB-D datasets and provide no monocular pipeline.
- Semantic masks are prerequisites. Arbitrary camera feeds need an upstream segmentation stage that produces labels compatible with the mapper.
- The evaluation is static and indoor. Outdoor scale, changing geometry and independently moving objects fall outside the reported benchmark scope.
- Compute grows with the map. GPU memory use and optimization time increase with the number of Gaussians, making target room size an important capacity-planning variable.
- Class labels do not guarantee instance identity. Applications that must distinguish several objects from the same class should verify the exported label format and add persistent instance association where needed.
Viewer compatibility needs checking
The pipeline saves Gaussian reconstructions as .ply files and provides an interactive viewer for appearance and color-coded semantic results. Standard 3DGS viewers can inspect compatible geometry and appearance attributes, while custom semantic fields may require SGS-SLAM’s visualization code or an adapter.
A base for semantic Gaussian SLAM
SGS-SLAM sits between photorealistic Gaussian SLAM and conventional semantic mapping, giving robotics and AR researchers one representation for camera pose estimation, rendering and class-aware geometry. Subsequent projects have extended this direction toward open-set labels and dynamic scenes; LVD-GS, for example, targets dynamic environments. The released preprocessing, optimization and viewing pipeline makes SGS-SLAM a reproducible baseline for indoor RGB-D experiments that need semantics integrated throughout the map.