Mistral AI's Mistral Small 3 Runs 3x Faster Than Llama 70B
Mistral drops a 24B Apache 2.0 model that matches Llama 3.3 70B in quality while running 3x faster on the same hardware.
Takeaways
- Mistral Small 3 is a 24B Apache 2.0 model matching Llama 3.3 70B quality at 3x the speed.
- Hits 81%+ MMLU accuracy and 150 tokens/s latency thanks to a shallower layer stack.
- Runs on a single RTX 4090 or a 32GB MacBook once quantized via Ollama.
- 32k context, Tekken tokenizer with 131k vocab, native function calling and JSON output.
- Deliberately released without RL or synthetic data, positioned as a base for reasoning fine-tunes.
- Available on Hugging Face, Ollama, Kaggle, Together AI, Fireworks, and la Plateforme.
Mistral AI has released Mistral Small 3, a 24-billion parameter model that punches well above its weight class and slots into a single RTX 4090 once quantized. The pitch is unusually specific: cover the 80% of generative AI tasks that need solid language understanding and instruction following, with latency low enough to feel instant.
Weights ship under Apache 2.0, as both a base checkpoint and an instruction-tuned version, and the model is already deployable through vLLM, Transformers, and Ollama.
The efficiency trick hiding in the architecture
The interesting engineering choice is architectural rather than algorithmic. Mistral Small 3 has far fewer layers than competing models, substantially reducing the time per forward pass. Fewer layers means fewer sequential matrix multiplications per token, which translates directly to lower latency without touching parallelism tricks or speculative decoding.
At over 81% accuracy on MMLU and 150 tokens/s latency, Mistral Small is currently the most efficient model of its category. On identical hardware, the team claims parity with Llama 3.3 70B Instruct while running more than three times faster.
One subtle but important detail: according to Mistral, this checkpoint was neither trained with reinforcement learning nor on synthetic data. That makes it a deliberately unfinished product, positioned as a clean base for the community to bolt reasoning capabilities onto, complementing releases like DeepSeek R1.