Unsloth Makes NVIDIA's Nemotron Ultra 550B Locally Runnable at 359GB
Unsloth releases GGUF-quantized versions of NVIDIA's 550B Nemotron 3 Ultra, making the fastest US open-weight frontier model runnable locally for the first time.
Takeaways
- Unsloth released GGUF-quantized versions of NVIDIA Nemotron 3 Ultra 550B, enabling local inference via llama.cpp, Ollama, and LM Studio.
- The model scores 48 on the Artificial Analysis Intelligence Index — the highest of any US-developed open-weight model — but trails China's Kimi K2.6 (54).
- Nemotron 3 Ultra achieves up to 5.9x higher inference throughput than comparable open models and lowers agentic task cost by up to 30%.
- Architecture is a novel Mamba-2 + MoE + Attention hybrid (LatentMoE) with Multi-Token Prediction, supporting up to 1M token context.
- Quantized GGUF sizes range from 188 GB (1-bit dynamic) to 359 GB (4-bit Q4_K_M); full BF16 requires 8×H100 or equivalent.
- Available free on OpenRouter and via API on DeepInfra ($0.37/M input, $1.08/M output); released under the permissive OpenMDW-1.1 license.
NVIDIA's largest open-weight model just got a lot more accessible. Unsloth has released GGUF-quantized versions of NVIDIA Nemotron 3 Ultra, a 550-billion-parameter reasoning model that NVIDIA announced at Computex 2026 and shipped on June 4. The GGUF release brings the model into the local inference ecosystem, with quantization levels ranging from 1-bit (189 GB) all the way to full BF16 (1.1 TB), and compatibility with llama.cpp, Ollama, LM Studio, vLLM, and SGLang.
The Biggest US Open Model, Now Locally Runnable
Nemotron 3 Ultra has 550 billion total parameters with 55 billion active per forward pass, uses a hybrid Mamba-Transformer MoE architecture, and scores 48 on the Artificial Analysis Intelligence Index , the highest score of any US-developed open-weight model as of June 2026. For most of 2026, the open-source leaderboard has been a Chinese competition , Kimi K2.6, GLM-5.1, Qwen 3.5 , while the US open-weight tier was stuck at 33–36 on the Intelligence Index. Nemotron 3 Ultra closes that gap meaningfully, though not entirely.
On Artificial Analysis's Intelligence Index it scores 48 (ninth of eighty-nine), ahead of all US open peers but trailing China's Kimi K2.6 at 54 , a six-point gap. It's the leading US-origin open-weight model, not the best open model overall. But where it genuinely stands out is speed.
Nemotron 3 Ultra achieves 5.9x, 4.8x, and 1.6x higher inference throughput compared to GLM-5.1-754B-A40B, Kimi-K2.6-1T-A32B, and Qwen-3.5-397B-17B respectively on the 8k token input / 64k token output setting. In experiments on SWE-bench and Terminal Bench 2.0, it completed benchmarks using fewer total tokens and fewer tokens per turn than comparable models, lowering the cost for agentic tasks by up to 30%.
A Novel Architecture Under the Hood
The model's efficiency story starts with its architecture. Architectural innovations include hybrid Mamba-Transformer layers for efficient long-context handling, NVFP4 quantization for cross-architecture GPU deployment, LatentMoE for expert routing, and multi-token prediction for improved generative speed in multi-turn tasks.
To unpack that: LatentMoE (Latent Mixture-of-Experts) is a routing mechanism where tokens are first projected into a smaller compressed space before being sent to experts, improving how much useful computation you get per byte of model. Multi-Token Prediction (MTP) means the model predicts multiple future tokens at once during training, which doubles as a native speculative decoding mechanism at inference time , generating drafts faster without a separate draft model. The MTP layers use a shared-weight design across prediction heads, which improves training signal quality, enables faster inference via native speculative decoding, and supports more stable autoregressive drafting at longer draft lengths.