Skip to main content
Back

Johns Hopkins' ColBERTSaR Slashes AI Search Index Size by 70% Without Retraining

ColBERTSaR converts ColBERT's bloated multi-vector index into a true inverted index, cutting storage 50–70% while keeping retrieval quality competitive

  • ColBERTSaR converts ColBERT's dense multi-vector index into a true sparse inverted index by discarding residual vectors, cutting storage 50–70% vs. 1-bit PLAID.
  • The method requires no model retraining and works as a drop-in replacement for PLAID with any existing ColBERT checkpoint.
  • Retrieval quality is retained at 89–92% of PLAID 1-bit effectiveness across BEIR benchmarks.
  • The paper proves ColBERT with residual-free quantization is theoretically equivalent to learned sparse retrieval (e.g., SPLADE), bridging two major IR paradigms.
  • Weakest on entity-heavy QA tasks (Quora, FEVER, NQ); BM25 fusion partially recovers performance at low extra cost.
  • Code is open-source at github.com/hltcoe/ColBERTSaR; paper accepted at SIGIR 2026.

ColBERT is one of the most effective neural retrieval architectures around, but it comes with a painful tax: its index is enormous. A team from Johns Hopkins University has published ColBERTSaR, a method that uses product quantization to transform ColBERT's dense multi-vector index into a true sparse inverted index, slashing storage by 50–70% compared to the already-compressed one-bit PLAID baseline, with no retraining required.

Why ColBERT's Index Is So Heavy

ColBERT's power comes from its late interaction mechanism: rather than compressing a document into a single embedding, it keeps one vector per token. At query time, it computes a MaxSim score, which finds the best-matching document token for each query token and sums those scores. This fine-grained matching is why ColBERT consistently outperforms single-vector models on retrieval benchmarks.

The cost is index size. For one million documents, each with 512 tokens, PLAID still takes more than 7.8 GB to store aggressive 1-bit compressed residuals for the default ColBERT embedding size of 128. That size doubles or quadruples with 2 or 4-bit compression. Indexes in PLAID and similar ColBERT implementations require five to ten times the disk storage of the original raw text, which limits their scalability.

The standard fix, PLAID, compresses each token embedding as a cluster centroid ID plus a small residual vector. PLAID uses centroid interaction as well as centroid pruning, a mechanism for sparsifying the bag of centroids, within a highly-optimized engine to reduce late interaction search latency by up to 7x on a GPU and 45x on a CPU against vanilla ColBERTv2. But PLAID only solves latency, not storage. Prior work has identified that the gathering and decompression stages are the primary inefficiencies at query time, and limiting the number of document tokens that must be gathered does not eliminate the need for the entire index to support ad hoc queries. Those residual vectors, while small, still compose the majority of the index.

The Core Insight: Drop the Residuals Entirely

ColBERTSaR's key observation is mathematical. In PLAID, each document token embedding is decomposed as a centroid plus a residual. If you simply discard the residual and score documents using only centroid assignments, the MaxSim scoring function collapses into something structurally identical to a sparse inverted index lookup. The authors show that, theoretically, ColBERT with embedding quantization is equivalent to learned-sparse retrieval except for the scoring mechanism. Using this residual-free product quantization, ColBERT is a learned sparse retrieval model with complex score aggregation.