Microsoft's Dion3 Makes the Optimizer Powering Kimi K2 Run 6x Faster
Microsoft Research's Dion3 optimizer slashes Muon's orthogonalization overhead by up to 6x through smarter math, custom kernels, and selective row updates.
Takeaways
- Microsoft Research released Dion3, a Muon optimizer replacement with up to 6x faster steps
- Available as pip install via the microsoft/dion package with FSDP2 and DDP support
- Gram Newton-Schulz reformulates orthogonalization to iterate on the small symmetric Gram matrix
- Custom CuteDSL kernels exploit matrix symmetry for roughly 2x speedup over cuBLAS
- Fractional row selection with error feedback surprisingly improved loss over tuned NorMuon at every scale tested
- Megabatching cuts communication rounds to O(1) regardless of model depth
Muon has quietly become the optimizer powering some of the largest language models being trained today, from Kimi K2 to GLM-5. It comes with a tax, though: an expensive Newton-Schulz orthogonalization step that gets painful as models grow. A team from Microsoft Research, Princeton, NYU, and NVIDIA just released Dion3, a drop-in Muon replacement that matches or improves on the loss Muon achieves while reducing optimizer step time by up to 6x. It ships today as part of the microsoft/dion package.

Why Muon needs a rescue
Muon works by taking the momentum matrix and running it through a polar decomposition, which rebalances the singular values of each update. It reaches a given loss in fewer steps than AdamW, but each step is heavier. The overhead comes from Muon's Newton-Schulz orthogonalization procedure, a cubic-time matrix operation not present in older optimizers. When weights are sharded across GPUs, Muon must gather the shards together to compute polar on the full matrix, piling on communication overhead that AdamW doesn't have.
Kimi's team made Muon scale, but the authors point out that success required a very specific alignment of architecture, parallelism strategy, and framework versions. Dion3 is designed to remove that fragility.
Four ideas that stack
Dion3 attacks the overhead at every layer of the stack. Each of the four contributions is useful alone, and they compound when combined:
- Gram Newton-Schulz. A mathematically equivalent reformulation that iterates on the small n by n Gram matrix instead of the full n by m weight matrix.
- Custom CuteDSL symmetric GEMM kernels. Exploit the symmetry of intermediate matrices to halve FLOPs on Hopper and Blackwell GPUs.
- Fractional update rule.