Gemini 3.1 Pro Ties Embedding Models but Costs 1,431x More
A controlled benchmark of 10 LLMs against 26 embedding models on 37 tasks finds parity in quality but a 1,431x cost gap.
Takeaways
- New COLM 2026 benchmark MTEB(LLM) pits 10 LLMs against 26 embedding models on 37 tasks.
- Best LLM (Gemini 3.1 Pro, 77.6) beats best embedder (77.2) by only 0.4 points overall.
- LLMs cost up to 1,431x more per benchmark pass: $154 versus $0.11.
- Open LLMs run 2.5 to 736 times slower per token than embedders on the same GPU.
- Reasoning tokens are 28 to 81 percent of LLM cost; lower budgets often help retrieval.
- Code, datasets, and figures are open source at embedders-dilemma.
If you have ever wondered whether to rip out your text-embedding pipeline and just ask an LLM to do the job, a new benchmark has the receipts. The Embedder's Dilemma, accepted to COLM 2026, runs a controlled bake-off between prompting frontier LLMs directly and using dedicated embedding models, and it accounts for every dollar and token spent along the way.
The headline finding is uncomfortable for both camps. In aggregate, the two paradigms tie: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Getting to that tie with an LLM is roughly a thousand times more expensive.
The bake-off nobody had run cleanly
Text embedding is the quiet workhorse behind retrieval-augmented generation, semantic search, clustering pipelines, and classification services. The last two years blurred the line between embedders and generators: GritLM, NV-Embed, and E5-Mistral all showed that decoder-only LLMs can be fine-tuned into strong embedders. If LLMs are already this good at semantic understanding, do you still need a separate embedding stack?
The authors, from the team behind MTEB, built MTEB(LLM) to answer that head-on. They evaluate ten LLMs across six families and 26 embedding models ranging from 118M to 14B parameters on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. Both sides see identical held-out data, and every run is priced.