Skip to main content
Back

Cohere Labs' Aya-23-8B Beats Mistral and Gemma Across 23 Languages

Cohere Labs' Aya-23-8B brings instruction-following quality to 23 languages, beating Mistral and Gemma in multilingual benchmarks with an open-weights 8B model

  • Aya-23-8B is an open-weights 8B instruction-tuned model from Cohere Labs supporting 23 languages, available on Hugging Face.
  • Beats Mistral-7B and Gemma-1.1-7B in multilingual benchmarks, with GPT-4-judged win rates of 65%+ across 10 languages.
  • Built on Cohere's Command R base with a 256k multilingual BPE tokenizer, GQA, SwiGLU, and RoPE; trained on ~10.5M multilingual instruction samples.
  • Strongest for translation, summarization, and open-ended generation in non-European languages like Arabic, Hindi, Turkish, and Japanese.
  • Free for research under CC-BY-NC; compatible with Transformers, vLLM, SGLang, llama.cpp, and Ollama.
  • Superseded by Aya Expanse 8B for new projects; use Aya-23-8B for reproducibility with the original 2024 technical report.

Aya-23-8B is an open-weights, instruction-tuned language model from Cohere Labs built to serve 23 languages at a quality level that most multilingual models have never reached. It sits at 8 billion parameters, runs on consumer-grade hardware, and consistently outperforms same-size models like Mistral-7B and Gemma-1.1-7B across translation, summarization, reasoning, and open-ended generation tasks in languages far beyond English.

The problem it actually solves

Most LLM progress has been English-first. Models like Mistral and Gemma are heavily used by multilingual communities, but they were never explicitly trained for it. The result is a steep performance cliff the moment you leave high-resource European languages. Multilingual efforts like Aya 101, BLOOMZ, and mT0 made strides in expanding access to NLP for the world, but significant room for improvement remained relative to first-class citizen languages like English and Chinese.

The previous Aya model (Aya 101) tried to solve this with sheer breadth, covering 101 languages. But breadth came at a cost. Aya 23 is an experiment in depth vs breadth, exploring the impact of allocating more capacity to fewer languages that are included during pre-training. The bet paid off.

What's under the hood

Aya-23-8B is a multilingual language model that supports 23 languages and builds on the Command family architecture, integrating the Aya Collection dataset. The base model is Cohere's Command R, which was pre-trained on data spanning all 23 target languages, giving the model a strong multilingual foundation before any fine-tuning begins.

The architecture uses several modern design choices from the Aya 23 technical report:

  • Grouped Query Attention (GQA): Multiple query heads share a single key-value head, cutting inference-time memory usage significantly.
  • SwiGLU activations: A gating mechanism in the feed-forward layers that consistently improves downstream task performance over standard ReLU.
  • Rotary Positional Embeddings (RoPE): Enables better generalization to longer contexts than alternatives like ALiBi.