NVIDIA's Nemotron 3 Super Hits 2.2x Faster Inference for AI Agents
NVIDIA's 120B hybrid Mamba-Transformer MoE with LatentMoE routing, 1M context, and NVFP4 pretraining lands as GGUFs runnable locally.
Takeaways
- NVIDIA released Nemotron 3 Super: 120B total, 12B active hybrid Mamba-Transformer MoE for agentic reasoning.
- New LatentMoE routes tokens in a compressed latent space; 512 experts, 22 active, beats standard MoE baseline.
- Multi-Token Prediction plus NVFP4 pretraining deliver up to 2.2x throughput vs GPT-OSS-120B and Qwen3.5-122B.
- Native 1M-token context, RULER-100 score of 91.75 at full length, toggleable reasoning trace.
- Available via build.nvidia.com, AWS Bedrock, OpenRouter free tier, and Unsloth GGUFs for local use.
- Self-host NIM needs 3x H100/B200/RTX PRO 6000; NVFP4 build runs on a single B200 or DGX Spark.
NVIDIA has released Nemotron 3 Super weights for a 120-billion-parameter, open-weight model that activates about 12 billion parameters per token. Unsloth has also published Unsloth GGUF builds for llama.cpp. The model combines Mamba-2, selective attention, compressed expert routing, and multi-token prediction to serve coding, tool use, retrieval, and other workloads that generate long traces across repeated model calls.
| Specification | Nemotron 3 Super |
|---|---|
| Total parameters | 120 billion |
| Active parameters | About 12 billion per token |
| Architecture | Hybrid Mamba-2, attention, and LatentMoE |
| Maximum context | 1 million tokens |
| Pretraining data | 25 trillion text tokens |
| Primary use cases | Agents, coding, tool calling, long-document retrieval, and high-throughput chat |
Three mechanisms cut inference cost
Nemotron 3 Super interleaves Mamba-2 and mixture-of-experts layers with a smaller number of attention layers. Mamba-2 processes sequences with state-space operations whose cost grows linearly with sequence length, making very long contexts more tractable. Selective attention layers retain the direct token-to-token lookup behavior associated with Transformers.
- LatentMoE compresses expert computation. Tokens are projected into a smaller latent representation before routing and expert processing. NVIDIA says this design can activate four experts for roughly the inference cost of one conventional expert.
- Multi-token prediction drafts ahead. Additional prediction heads propose several future tokens during one model step. A compatible serving engine can verify those drafts through native speculative decoding and emit more than one token at a time.
- Sparse activation limits per-token work. Each token uses about 12 billion parameters, while storage still covers the full 120-billion-parameter expert pool. Local deployments therefore need memory for the selected quantization of the complete checkpoint.
An ablation in NVIDIA’s technical report compared a LatentMoE configuration with a 1,024-dimensional latent space, 512 total experts, and 22 active experts against a conventional MoE with 128 total experts and six active experts. The LatentMoE variant performed better at matched parameter and token budgets, supporting NVIDIA’s claim that compressed routing improves accuracy per parameter and per unit of computation.
Multi-token prediction only accelerates generation when the inference stack consumes the extra heads. Runtimes that load the base model but ignore those heads will miss that portion of the advertised throughput gain.