Nomic AI's nomic-embed-text-v2-moe Beats 500M Models at 300M Active Parameters
Nomic ships the first sparse Mixture of Experts general purpose text embedding model, delivering multilingual retrieval quality at half the active parameters.
Takeaways
- Nomic released nomic-embed-text-v2-moe, the first general purpose MoE text embedding model, under Apache 2.0.
- Architecture: 475M total parameters, 305M active, 8 experts with top-2 routing, 512 token context.
- Scores 52.86 BEIR and 65.80 MIRACL, beating same-size dense peers and competing with 568M models.
- Trained on 1.6B multilingual pairs across ~100 languages using contrastive pretraining plus supervised finetuning.
- Matryoshka embeddings let you truncate 768d vectors to 256d for roughly 3x storage savings.
- Full weights, training code, and data available via the contrastors repo.
Embedding models have quietly been growing bigger to chase leaderboard scores, which becomes a problem when you need to index millions of documents into a vector database and keep query latency under control. Nomic AI is trying to break that trade-off with nomic-embed-text-v2-moe, a multilingual retrieval model that uses a sparse Mixture of Experts (MoE) architecture to keep inference costs closer to a small model while matching the quality of much larger ones.
It covers around 100 languages, was trained on over 1.6 billion pairs, and uses Matryoshka embeddings so you can truncate vectors down to a third of their size with minimal quality loss. The weights are Apache 2.0 licensed, and the training code and data pipeline are public.
MoE, finally applied to encoders
Mixture of Experts is standard in generative LLMs, but it hadn't been successfully adapted to general text embeddings before. Nomic Embed v2 is the first general purpose MoE text embedding model. The architecture uses alternating MoE layers with 8 experts and top-2 routing, giving 475M total parameters but only 305M active during training and inference.
Because only two experts fire per token, you get the representational capacity of a bigger model while paying the compute bill of a roughly 300M dense model. That matters for RAG workloads, where a larger model's memory footprint constrains ingestion throughput and its higher latency shows up at query time.