Hackers Can Trick AI Code Reviewers Into Approving Vulnerable Code 97% of the Time
A new study shows adversarial pull request metadata can slip known CVEs past LLM code reviewers like Claude Code and CodeRabbit in 97% of cases.
Takeaways
- New ICSE 2027 paper shows LLM code reviewers suffer from framing bias exploitable as a supply-chain attack.
- Framing a diff as bug-free cuts vulnerability detection by 16-93% across four frontier models, with false negatives spiking.
- Injection flaws are far more susceptible to framing bias than memory corruption bugs.
- A novel LLM-assisted refinement attack slips 32 of 33 CVEs past Claude Code and CodeRabbit, a 97% success rate.
- Attackers exploit an asymmetry: unlimited offline iteration against a local clone versus one defender detection shot.
- Template-based bias attacks backfire; metadata redaction and debiasing prompts help but do not fully close the gap.
Pull-request titles, descriptions, commit messages, and labels can reduce an LLM reviewer’s ability to detect vulnerable code, according to a new paper from researchers at the Athens University of Economics and Business and Delft University of Technology. Their experiments show that attackers can exploit this framing effect to reintroduce known vulnerabilities while presenting the change as a routine fix, refactor, or dependency update.
Automated code review systems increasingly combine source diffs with contributor-written context before commenting on or approving a change. That context helps a model understand intent, but it also gives untrusted prose influence over the security verdict. The researchers call this attack technique bias injection.
Framing changes the verdict
The first study measured whether presentation could override evidence in the code. The exploratory study was conducted across 6 LLMs under five framing conditions. Describing a change as bug-free reduced vulnerability detection by a reported 16% to 93%, depending on the model.
The errors were concentrated in false negatives, where vulnerable code was classified as safe. False-positive rates, where safe code was flagged as vulnerable, changed little. Injection flaws were more susceptible to framing than memory-corruption bugs, suggesting that exploit class affects how readily contextual claims sway a model.
The latest paper also includes a broader analysis across six LLMs and five framing conditions. Those results found the same pattern across multiple model families, indicating that the effect extends beyond one provider or review configuration.