NVIDIA's Cosmos 3 Unifies Robot Reasoning, Video, and Action in One Open Model
NVIDIA's Cosmos 3 fuses vision reasoning, video generation, and robot action prediction into one open omnimodal world model for physical AI.
Takeaways
- NVIDIA released Cosmos 3, an open omnimodal world model unifying reasoning, video generation, and robot action prediction.
- Two checkpoints on Hugging Face: Nano (16B) and Super (64B); Edge variant coming for real-time inference.
- Mixture-of-Transformers architecture pairs an autoregressive VLM reasoner with a diffusion generator sharing 3D mRoPE positional embeddings.
- Trained on 20 trillion multimodal tokens including 1B images, 400M videos, audio, and human plus robot action data.
- Leads open-source leaderboards on R-Bench, Physics-IQ, RoboLab, VANTAGE-Bench, and TAR for vision understanding.
- Runs via Diffusers for research or vLLM-Omni for OpenAI-compatible production serving under OpenMDW license.
NVIDIA just dropped a swing at one of the hardest problems in robotics: teaching machines to predict what happens next in the messy physical world. Cosmos 3 is an open omnimodal world model that can watch a video and reason about it, generate a physically plausible future, and even spit out the joint angles a robot would need to act on that prediction, all from the same network.
NVIDIA released Cosmos 3, a family of omnimodal world models for physical AI that combine physical reasoning, world generation, and action generation inside one open model. The release ships as two open-weight checkpoints on Hugging Face: a Super version with 64.6 billion parameters and a Nano version with 15.7 billion parameters, both available with no gatekeeping, with an Edge variant for real-time on-device inference coming later.
One model, three jobs
Earlier Cosmos releases kept perception and generation as separate models. Cosmos 3 collapses them. It uses a Mixture-of-Transformers architecture built around two towers: a reasoner tower that is a vision-language model interpreting images, video, and text autoregressively to understand motion, object interactions, and physical context, serving as the brain that reasons about the world before any generation happens. The second tower is a diffusion-based path for physics-aware video and actions, conditioned on the reasoner's understanding, with information flowing one way from reasoner to generator, and both towers share a 3D multimodal RoPE.
That split shows up in the two runtime surfaces developers actually use:
- Reasoner: text and vision in, text out. Captioning, temporal localization, embodied reasoning, 2D grounding, physical plausibility classification.
- Generator: text, vision, sound, or action in; video, sound, or action out. Synthetic training data, forward and inverse dynamics, robot policy rollouts.
The headline trick is that the model treats action as a core modality with dedicated action tokens, accepting text, image, video, and JSON action arrays as inputs and producing images, video, synchronized sound, action states, and text. Action conditioning spans camera, vehicle, egocentric, single-arm, dual-arm, and humanoid embodiments, with each embodiment using a fixed action dimension such as 9D for cameras.