Cohere's North Mini Code 1.0 Beats 120B Models With Just 3B Active Parameters
Cohere's North Mini Code is a 30B MoE model with only 3B active parameters that beats 120B+ models on agentic coding benchmarks, free under Apache 2.0
Takeaways
- North Mini Code 1.0 is a 30B MoE model with only 3B active parameters, beating 120B+ models on Artificial Analysis' Coding Index with a score of 33.4.
- Released under Apache 2.0, available free on Hugging Face in BF16 (Transformers) and FP8 (vLLM) variants.
- Achieves 2.8x higher output throughput and 30% better inter-token latency vs comparable dense models in Cohere's internal tests.
- Trained with two-stage SFT + RLVR on 70k+ verifiable tasks across ~5k unique repositories, with explicit multi-harness robustness training.
- Supports a 256K context window with 64K max output, designed for repo-level agentic coding, terminal automation, and local private deployment.
- First model in Cohere's new North family; works natively with OpenCode, SWE-Agent, and most coding agent frameworks.
Cohere just released North Mini Code 1.0, an open-weight coding model that challenges the assumption that bigger always means better. The model has 30 billion total parameters, but only 3 billion are active at any given moment. That architectural trick, called a sparse Mixture-of-Experts (MoE), lets it deliver large-model quality at a fraction of the compute cost.
The headline numbers are hard to ignore. On Artificial Analysis' Coding Index, North Mini Code achieves a score of 33.4, outperforming Qwen3.5 (35B-A3B), Gemma 4 (26B-A4B), Devstral Small 2 (24B Dense), and even substantially larger models such as Nemotron 3 Super (120B-A12B), Mistral Small 4 (119B-A6B), and Devstral 2 (123B). A model with 3B active parameters beating models with 10-40x more compute per token is not the norm.
The MoE trick that makes it fast
Sparse MoE works by splitting the model's feed-forward layers into many specialized sub-networks called "experts." Instead of running every parameter on every token, a router picks a small subset of experts for each token. The feed-forward block uses 128 experts, of which 8 are activated per token. Each expert block is an FFN block with SwiGLU activation, and the router applies a sigmoid activation function to the logits before the top-k selection. You get the capacity of a 30B model with the inference cost closer to a 3B one.
The attention mechanism is also worth noting. North Mini Code is a decoder-only Transformer-based sparse Mixture-of-Experts model that uses an efficient attention implementation, interleaved between sliding-window attention with RoPE and global attention with no positional embeddings, in a 3:1 ratio. This hybrid approach helps the model handle both local code structure and long-range dependencies across a massive context window.
The throughput gains are real. In Cohere's internal tests, North Mini Code reached up to 2.8x higher output throughput at identical concurrency and hardware, and showed a 30% edge in inter-token latency.
How it was trained
North Mini Code was post-trained using a two-stage cascaded supervised fine-tuning (SFT) followed by reinforcement learning with verifiable rewards (RLVR), focusing on agentic coding. RLVR is a training technique where the model gets a reward signal based on whether its outputs are verifiably correct, rather than relying on human preference ratings. Think of it as training the model by actually running its code and checking if the tests pass.