Cohere Labs' Tiny Aya Beats Gemma on 46 Languages at 3B Parameters
Cohere Labs ships a 3.35B multilingual model family covering 70+ languages, with regional variants that beat larger competitors on low-resource translation.
Takeaways
- Cohere Labs released Tiny Aya, a 3.35B multilingual model family covering 70+ languages.
- Five checkpoints: Base, Global, and regional variants Earth, Fire, and Water.
- Beats Gemma3-4B on average Flores and WMT24++ translation, wins 46 of 61 WMT24++ languages.
- Scores 39.2% on African-language MGSM versus Gemma3-4B's 17.6% and Qwen3-4B's 6.25%.
- Regional models built via SimMerge parameter-space merging, up to +5.5 ChrF gain in South Asia.
- Trained on 64 H100s, available on Hugging Face, Kaggle, and Ollama.
Multilingual language models tend to be either huge and English-heavy, or small and narrow. Cohere Labs is pushing on a third path with Tiny Aya, a family of compact open-weight models built to run on a laptop while genuinely serving speakers of languages that usually get scraps of training data. Tiny Aya is a 3.35 billion-parameter multilingual open-weight model designed to deliver high-quality translation and language understanding while running on local and consumer-grade hardware.
The release is not a single model but a small ecosystem. The Tiny Aya family includes TinyAya-Base, which covers more than 70 languages, and TinyAya-Global, an instruction-tuned model supporting 67 languages with balanced performance across regions. Cohere is also introducing three regionally specialized variants, TinyAya-Earth, TinyAya-Fire and TinyAya-Water, optimized for Africa and West Asia, South Asia, and Asia-Pacific and Europe, respectively. All five are on Hugging Face under a research license, along with a fine-tuning dataset and technical report.
The uneven multilingual problem
Progress in multilingual modeling has been lopsided because performance tracks how much web text exists in each language. Frontier models are trained largely on English and a handful of high-resource European and East Asian languages, which leaves speakers of Hausa, Amharic, Marathi, or Khmer with translation systems that hallucinate script, drop diacritics, or silently switch back to English mid-sentence. Existing small open models like Gemma 3, Qwen 3, SmolLM 3, and Ministral cover a limited language set and, at 3 to 4 billion parameters, have to trade breadth for depth.
Tiny Aya's premise is that this trade-off is partly a data and training-recipe problem, not just a parameter-count problem. The models were trained on a single cluster of 64 H100 GPUs using relatively modest computing resources, which makes the recipe reproducible and iterable rather than dependent on hyperscaler budgets.
Architecture and tokenizer choices
Under the hood, Tiny Aya inherits design ideas from Cohere's larger Command A models. It uses parallel transformer blocks, interleaves three sliding-window attention layers (window size 4096, with rotary positional embeddings) with one full-attention layer that uses no positional embeddings at all, and applies SwiGLU activations with grouped-query attention to keep inference memory in check. The base model is pretrained on roughly 6 trillion tokens using a warmup-stable-decay learning rate schedule, with an 8K context window.