Mistral's Mixtral-8x7B Beats GPT-3.5 With 6x Faster Open-Source Inference
Mistral's sparse mixture-of-experts model keeps trending on Hugging Face, offering GPT-3.5 level quality with only 13B active parameters per token.
Takeaways
- Mixtral-8x7B-Instruct remains a top-trending Hugging Face model with 4,700+ likes and 338k downloads.
- Sparse MoE routes each token through 2 of 8 experts, activating only 13B of 47B parameters.
- Delivers ~6x faster inference than Llama 2 70B while matching or beating GPT-3.5 on benchmarks.
- Apache 2.0 licensed, 32k context, and native support for English, French, Italian, German, Spanish.
- Instruct variant tuned with SFT and DPO, beating GPT-3.5 Turbo and Claude-2.1 on human evals.
- Available via Hugging Face, NVIDIA NIM, and Ollama.
Mistral's Mixtral-8x7B-Instruct-v0.1 continues to sit near the top of the Hugging Face model charts, racking up hundreds of thousands of monthly downloads long after its debut. The instruction-tuned sparse mixture-of-experts model has become a default open-weight choice for teams that want strong reasoning without paying for a frontier API, thanks to an architectural trick that decouples parameter count from inference cost.
The routing trick behind the numbers
The naming misleads a little. An 8x7B MoE model lands at roughly 45B parameters because the mixture-of-experts layer duplicates the feedforward network eight times while leaving attention unitary on each layer. So the model is not stacking eight full 7B copies, but eight FFN blocks plus a router and shared attention weights.
Sparse activation is what makes inference fast. Mixtral inherits the Mistral 7B architecture, except each layer contains eight feedforward experts. For every token at each layer, a router picks two experts, runs the hidden state through them, and combines their outputs. The chosen pair can change at every timestep, so a token has access to 47B parameters overall while touching just 13B during any given forward pass. That top-2 routing is why Mixtral outperforms Llama 2 70B on most benchmarks with roughly 6x faster inference.