Skip to main content
Back

NeuML's PubMedBERT Beats General-Purpose Encoders on Medical Search

A compact biomedical embeddings model fine-tuned on PubMed abstracts is outperforming general-purpose encoders on medical text, at only 100M parameters.

  • NeuML/pubmedbert-base-embeddings is a 100M parameter biomedical sentence encoder fine-tuned from Microsoft BiomedBERT.
  • Beats bge-base, gte-base and MiniLM on three PubMed benchmarks with a 95.62 average Pearson score.
  • Trained one epoch on PubMed title-abstract pairs using MultipleNegativesRankingLoss in sentence-transformers.
  • Outputs 768-dim vectors with 512 token context, mean-pooled, Apache 2.0 licensed.
  • Powers 59+ Hugging Face Spaces including medical RAG chatbots and clinical document QA systems.
  • Integrates natively with txtai for embeddings databases and retrieval augmented generation.

At publication, Hugging Face showed close to one million downloads for NeuML’s model, a 100 million-parameter encoder built for biomedical text. It converts sentences and passages into 768-dimensional vectors for semantic search, clustering, and retrieval-augmented generation (RAG). In author-reported results, it scored slightly above widely used general-purpose encoders across three PubMed-derived evaluations.

The model fine-tunes Microsoft BiomedBERT, formerly PubMedBERT, through the Sentence Transformers framework. BiomedBERT was pretrained on biomedical literature, while this additional training teaches it to place related passages near one another in vector space. Applications can then rank documents by cosine similarity between their vectors.

A narrow but consistent benchmark lead

The model card reports higher scores than bge-base-en-v1.5, gte-base, all-MiniLM-L6-v2, and the medical retrieval model S-PubMedBert-MS-MARCO. Against the strongest comparison, gte-base, NeuML’s encoder leads by 0.30 points on PubMed QA, 0.10 on the PubMed subset, and 0.34 on PubMed Summary.

Author-reported scores; higher is better.
Model PubMed QA PubMed subset PubMed Summary Average
all-MiniLM-L6-v2 90.40 95.92 94.07 93.46
bge-base-en-v1.5 91.02 95.82 94.49 93.78
gte-base 92.97 96.90 96.24 95.37
pubmedbert-base-embeddings 93.27 97.00 96.58 95.62
S-PubMedBert-MS-MARCO 90.86 93.68 93.54 92.69

The 0.25-point average lead over gte-base is modest, and the published results include no confidence intervals or independent replication. They still make the model a credible candidate for PubMed-style retrieval, especially when deployment cost favors a compact encoder with a 512-token input limit.