Skip to main content
Back

Oracle's RASC Framework Beats GPT-4o by 3x on Medical Code Automation

A retrieve-then-classify pipeline beats zero-shot GPT-4o on clinical value set construction, turning an intractable generation problem into a shortlist classification task.

  • RASC beats zero-shot GPT-4o on clinical value set authoring by retrieving similar sets, then classifying candidate codes.
  • Cross-encoder fine-tuned on SAPBert hits F1 0.298 versus GPT-4o at 0.105 across 11,803 VSAC value sets.
  • Zero-shot GPT-4o hallucinated codes absent from VSAC at a 48.6% rate, confirming LLMs cannot memorize structured vocabularies.
  • Performance gap widens with set size: GPT-4o drops to F1 0.014 on sets over 150 codes; classifiers scale up.
  • Giving GPT-4o the retrieved pool triples F1 (0.089 to 0.294) but costs roughly 100x more than the MLP.
  • Code and benchmark pipeline released for reproducible research on corpus-grounded set completion.

Building a clinical value set, the enumerated list of standardized codes that pins down a concept like Type 2 Diabetes Without Complications, is one of those unglamorous tasks that quietly gates a lot of healthcare analytics. A new paper from Oracle Health Data Intelligence proposes Retrieval-Augmented Set Completion (RASC), a two-stage framework that reframes value set authoring as retrieval followed by binary classification and beats zero-shot GPT-4o by a wide margin on an 11,803-set benchmark drawn from the Value Set Authority Center.

When a spreadsheet job becomes an ML problem

Value sets are the plumbing of clinical quality measurement and phenotyping. The Value Set Authority Center (VSAC) is a tool suite from the National Library of Medicine that supports the creation and maintenance of these lists, which draw codes from standard vocabularies like SNOMED CT, RxNorm, CPT, and LOINC to define clinical concepts for interoperable health information exchange. Authoring them today is largely manual: a clinical informaticist searches a vocabulary of 100,000 to 1,000,000 codes and hand-picks the ones that belong.

The scale problem is getting worse. VSAC now hosts over 10,000 sets, and creation is accelerating sharply, reaching roughly 1,750 value sets per year in 2023 and 2024, more than the entire period from 2012 to 2020 combined. That growth turns manual curation into a bottleneck worth automating.

Where the obvious approach breaks

The natural first move is to ask an LLM to just list the codes. The authors tried this with GPT-4o on the full test set and got a value-set-level F1 of only 0.105. More striking: 48.6% of the codes GPT-4o returned did not exist in VSAC at all. Clinical code identifiers are structured, version-controlled strings, not natural language, and they are not reliably memorized during pretraining. The model confidently invents plausible-looking identifiers.