Skip to main content
Back

Answer.AI's answerai-colbert-small Beats 110M Models With Just 33M Parameters

Answer.AI's 33M-parameter ColBERT retriever beats models 3x its size on BEIR, runs on CPU, and slots into any existing ColBERT pipeline.

  • Answer.AI released answerai-colbert-small-v1, a 33M-parameter ColBERT retriever under Apache 2.0.
  • Scores 53.79 on BEIR average, beating bge-base-en-v1.5 (3.3x larger) at 53.25.
  • Outperforms the original ColBERTv2.0 across nearly every BEIR dataset with under a third of the parameters.
  • Runs on CPU with millisecond latency over hundreds of thousands of documents.
  • Works with sentence-transformers 6.0 MultiVectorEncoder, RAGatouille, rerankers, and Stanford colbert-ai.
  • Weaker on argument mining, short-query tasks like SCIDOCS, and duplicate detection (Quora, CQADupstack).

Answer.AI has released answerai-colbert-small-v1, a 33-million-parameter multi-vector retriever designed for fast CPU search. Answer.AI reports millisecond-scale retrieval across hundreds of thousands of documents and higher scores than the 110-million-parameter ColBERTv2 model on every benchmark it evaluated, including the training-unseen LoTTE suite.

Released under the Apache 2.0 license, the model can be downloaded and adapted without licensing fees. Hugging Face displayed more than 866,000 downloads at publication, although download counts do not indicate the number of production deployments. The training method extends Answer.AI’s earlier JaColBERT recipe for Japanese retrieval.

Why 96-dimensional token vectors help

Most dense retrieval models compress each document into one vector, which an approximate nearest-neighbor index can search efficiently. ColBERT-style retrieval preserves a vector for each token. For every query token, its MaxSim scoring function finds the most similar document token and then sums those maximum similarity scores.

Token-level scoring can preserve exact terms and local semantic matches that disappear when a document is reduced to one vector. It also creates more vectors per document, increasing index size and scoring work. Single-vector models remain common in retrieval-augmented generation systems because their indexes are simpler and usually smaller.

Answer.AI’s reported CPU latency suggests that a small backbone and compact 96-dimensional vectors can make this trade-off practical. Actual latency will vary with document length, index configuration, hardware, thread count, corpus size, and target recall.

A narrow BEIR lead, larger dataset gains