Valeo.ai's IGLOSS Teaches LiDAR to Spot Unseen Objects Without Camera Data
Valeo.ai's IGLOSS swaps CLIP text embeddings for text-to-image prototypes, setting new open-vocabulary segmentation records on nuScenes and SemanticKITTI.
Takeaways
- IGLOSS achieves state-of-the-art open-vocabulary lidar segmentation on nuScenes and SemanticKITTI.
- Replaces CLIP text embeddings with text-to-image prototype pictures to dodge the image-text modality gap.
- Uses a 3D backbone (ScaLR+) distilled from DINOv2 rather than from a VLM.
- Inference matches 3D point features to 2D prototype features via test-time multinomial logistic regression.
- Also sets SOTA on annotation-free closed-set segmentation on both datasets.
- Code, pretrained weights, and generated images released at valeoai/IGLOSS.
Valeo.ai researchers propose IGLOSS, a system that uses generated class images for open-vocabulary semantic segmentation of LiDAR point clouds. The NeurIPS 2026 paper reports state-of-the-art results on nuScenes and SemanticKITTI, while the released GitHub repository includes code, weights, and prototype images.
IGLOSS moves class matching into a DINOv2-derived visual feature space shared by generated images and 3D point features. This design removes the vision-language text encoder from the matching stage and lets the deployed model process LiDAR without synchronized camera images or scan sequences.
Text embeddings are the weak link
Open-vocabulary semantic segmentation assigns a class to every pixel or 3D point while allowing the evaluation vocabulary to include categories absent from model training. A common pipeline distills a vision-language model such as CLIP into a 3D backbone, then compares each point feature with text embeddings for candidate class names.
CLIP aligns images and text in a shared vector space, but the two modalities retain different feature distributions. That modality gap can reduce accuracy when image-trained representations are transferred to sparse 3D sensor data. Prompt wording adds another variable, with phrases such as “a photo of a truck” and “a LiDAR scan of a truck” producing different class vectors.