NVIDIA's Cosmos3-Edge Squeezes a Robot Brain Into a 4B On-Device Model
NVIDIA drops a 4B-parameter open world model that runs on Jetson, generating 32 robot actions per inference at 15 Hz control rates.
Takeaways
- Cosmos3-Edge is a 4B-parameter open omnimodal world model for physical AI at the edge.
- Combines an autoregressive tower and a diffusion tower sharing multimodal attention layers.
- Handles text, image, video, and action trajectories for 10+ robot embodiments.
- Runs on Jetson Thor at 15 Hz control with 32 actions per inference.
- Ranks #1 among 4B models on VANTAGE-Bench and SOTA for robot policy learning.
- Released under OpenMDW1.1 with vLLM, Diffusers, and PyTorch support.
NVIDIA has released Cosmos3-Edge, a 4-billion-parameter open world model built for on-device physical AI. It targets the awkward middle ground where robots and vision agents need data-center-class multimodal reasoning but only have a Jetson module in the chassis. The model already has close to a million downloads on Hugging Face, and it ships under the permissive OpenMDW1.1 license for commercial and research use.
The pitch: one compact model that can watch a scene, reason about it in natural language, predict what happens next as video, and emit robot actions. It operates at robot-control resolution (640×360 observations), generating 32 actions per inference on NVIDIA Jetson Thor while sustaining real-time control at 15 Hz.
Two towers, one brain
The architecture is a Mixture-of-Transformers (MoT) with two specialized transformer stacks that share attention layers. An autoregressive transformer handles discrete token generation, while a diffusion transformer handles continuous multimodal generation. Text comes out through next-token decoding; images, video, audio, and actions come out through iterative denoising. The clever bit is that the two towers keep separate normalization layers and MLPs but share multimodal attention layers, which align information across language, video, audio, and action.
Attention behavior is also modality-aware. Language uses causal attention, where each token attends to the tokens that came before it, while diffusion tokens attend more broadly to the available context, supporting coherent prediction and generation. In practice this means Cosmos3-Edge can chain a reasoning trace and a generated video rollout inside the same forward pass, without bolting together separate VLM and video-diffusion pipelines.
Actions as first-class citizens
Most video models treat actions as an afterthought. Cosmos3-Edge treats them like any other modality. Physical systems describe motion differently: a vehicle uses ego pose, a camera uses camera motion, a robot arm uses end-effector pose. Cosmos 3 maps these embodiments into a common representation encoding translation, rotation, and manipulation state as compact geometric vectors.