Mistral AI and NVIDIA Ship Mistral-Nemo With 128k Context on Apache 2.0
A 12B open-weight model from Mistral and NVIDIA arrives with a 128k context window, a new Tekken tokenizer, and Apache 2.0 licensing.
Takeaways
- Mistral AI and NVIDIA released a 12B instruction-tuned open-weight model under Apache 2.0 license
- Features a 128k token context window and quantization-aware training for FP8 inference with no accuracy loss
- New Tekken tokenizer is 30% more efficient on code and major European languages
- Strong multilingual coverage across 11+ languages including Korean, Arabic, and Hindi
- Drop-in replacement for Mistral 7B with function calling built into the chat template
- Available on Hugging Face, la Plateforme, NVIDIA NIM, and Ollama
Mistral AI and NVIDIA jointly shipped Mistral-Nemo-Instruct-2407, a 12B parameter instruction-tuned model that behaves like a straightforward upgrade path for anyone still running Mistral 7B. It offers a context window of up to 128k tokens, and its reasoning, world knowledge, and coding accuracy are competitive with anything in its size class. The weights are open under Apache 2.0, and the model can be pulled from Hugging Face, la Plateforme, NVIDIA NIM, Ollama, or Docker Hub.
Mistral brought the training recipe and post-training alignment, NVIDIA brought the compute and the NeMo framework tooling, and the result relies on standard architecture, making it a drop-in replacement in any system already using Mistral 7B. There are no new tricks in the transformer block, just more capacity and a much longer memory.
Under the hood
The architecture is a conventional decoder-only transformer with grouped-query attention. The specifics:
- 40 layers, hidden dim 5,120, head dim 128, MLP hidden dim 14,336
- 32 attention heads, 8 KV heads (GQA for cheaper long-context inference)
- SwiGLU activations and rotary position embeddings with theta = 1M
- Vocabulary of ~128k tokens using the new Tekken tokenizer
- Quantization-aware training, so you get FP8 inference with no accuracy loss
Training ran from June 2024 to July 2024, with pretraining data cut off at April 2024. The instruct checkpoint is a fine-tune of the separately released Mistral-Nemo-Base-2407.
Tekken, the tokenizer doing the heavy lifting
The most interesting engineering choice is the tokenizer swap. Mistral NeMo uses Tekken, a new tokenizer based on Tiktoken and trained on more than 100 languages. It compresses natural language and source code more efficiently than the SentencePiece tokenizer used in previous Mistral models.