Hochschule Esslingen Hits a Hard 0.30 Accuracy Wall Running Vul-RAG Locally
A new reproducibility study of Vul-RAG finds the framework works without proprietary APIs, but every open-weight model tested, including reasoning models, hits a hard ceiling of ~0.30 pairwise accuracy on vulnerability detection.
Takeaways
- A reproducibility study of Vul-RAG confirms the framework works fully locally with open-weight models, no proprietary APIs needed.
- All tested models, including reasoning models like QwQ-32B and DeepSeek-R1, hit a hard ceiling of ~0.30 pairwise accuracy on vulnerability detection.
- Pairwise accuracy measures whether a model correctly classifies both the vulnerable and patched version of the same code, a strict and meaningful metric.
- Reasoning models slightly outperform standard models but at 2.6x the token output cost, with no meaningful accuracy gain over the ceiling.
- Code-specialized models offer no significant advantage over general-purpose models of the same size, since Vul-RAG abstracts code into natural language descriptions.
- All code and evaluation artifacts are publicly available on GitHub for local deployment.
Security teams have long hoped that LLMs could automate the tedious work of hunting software vulnerabilities. Vul-RAG, a RAG-based vulnerability detection framework published last year, made a compelling case: by grounding LLMs with structured knowledge distilled from historical CVEs, it pushed pairwise accuracy well beyond what raw LLMs could achieve. The catch? Its strongest results relied on proprietary models like GPT-4o and Claude, making independent verification nearly impossible. A new study from researchers at Hochschule Esslingen sets out to fix that, and what they find is both reassuring and sobering.
The reproducibility problem hiding in plain sight
For approaches relying on proprietary models and APIs, reproducibility and replicability remain largely unexplored, raising the question of whether reported results generalize or depend primarily on specific model choices. This is not a minor footnote. If a security tool only works when you pipe code through a closed commercial API, you cannot audit it, run it air-gapped, or trust that results will be consistent as the underlying model silently updates. The ML community has increasingly recognized this gap, but vulnerability detection, a domain with real security stakes, has been slow to catch up.
The original Vul-RAG paper had already shown that LLMs have severe limitations in distinguishing between vulnerable and similar-but-benign patched code, with baseline accuracy of only 0.06 to 0.14. Vul-RAG addressed this by building a structured knowledge layer on top of raw LLM inference, and the gains were real. But reproducing those gains without GPT-4o access was an open question.
How Vul-RAG actually works
Vul-RAG works in three phases: first, it constructs a vulnerability knowledge base by extracting multi-dimensional knowledge via LLMs from existing CVE instances; second, for a given code snippet, it retrieves relevant vulnerability knowledge from the knowledge base based on functional semantics; third, it leverages LLMs to check the vulnerability of the given code snippet by reasoning the presence of vulnerability causes and fixing solutions of the retrieved vulnerability knowledge.