Researchers Strip Qwen2.5-Coder's Safety to Inject Real Security Flaws
A new paper surgically removes safety refusals from code LLMs to generate labeled vulnerable code for security research, revealing that willingness and capability are entirely separate properties.
Takeaways
- Researchers used abliteration -- a weight-level edit that removes a model's refusal direction -- to make safety-aligned code LLMs inject SQL vulnerabilities into safe code.
- The paper tests Qwen2.5-Coder at 3B, 7B, and 14B; the 14B refused 100% of prompts before abliteration, dropping to near-zero after.
- Post-abliteration injection success rates: 88-97% for 14B, 89-90% for 7B, but only 25-48% for 3B -- proving capability and willingness are separate properties.
- The goal is scalable, labeled vulnerable code generation for training vulnerability detectors, using CWE-89 (SQL injection) in Python as a case study.
- Syntactic validity stayed above 93% after abliteration, and outputs were verified by a CodeQL + Semgrep + Bandit ensemble with manual adjudication.
- This is a preliminary study: only one language, one CWE, and one model family tested; broader generalization is left for future work.
Building a reliable dataset of vulnerable code is one of the most stubborn bottlenecks in software security research. You need thousands of labeled examples to train vulnerability detectors, but real-world corpora are noisy, and asking an LLM to mutate existing vulnerable code just propagates the same errors. The cleaner approach -- starting from safe code and asking a model to inject a specific, well-defined vulnerability -- runs straight into a wall: safety-aligned code LLMs simply refuse. A new paper titled "Willing but Unable" asks whether that wall can be surgically removed, and what's left on the other side.
The answer is yes, the wall can come down -- but what you find behind it depends heavily on how big the model is. The paper is a feasibility study of abliteration applied to code LLMs, and its most important finding is a clean separation between two things that are often conflated: a model's willingness to do something, and its actual capability to do it correctly.
The Labeling Problem That Won't Go Away
Training a machine learning vulnerability detector requires a large corpus of code samples with accurate labels -- vulnerable or not. PromSec iteratively refines the task prompt based on feedback from a vulnerability scanner on the generated code , while SVEN and SafeCoder are fine-tuning-based methods that construct a training dataset containing code snippets with and without vulnerabilities, used to fine-tune the model. These approaches are effective but all share a common dependency: they need good labeled data to start with, or they need an LLM that will cooperate in generating it.
The alternative the authors explore is more direct: take a piece of safe code, give an instruction-tuned LLM a precise specification of a vulnerability (a CWE -- Common Weakness Enumeration type), and ask it to inject that flaw. This shifts the hard problem from open-ended detection to a simpler binary confirmation: does the output contain the specified bug? The catch is that safety-aligned models refuse these prompts outright.
What Abliteration Actually Does
Abliteration is a technique for removing refusal behaviors from language models by identifying and intervening on "refusal directions" in activation space, notionally represented via a single mean refusal direction. Think of it as a geometric operation on the model's internal state. Arditi et al. showed that refusal behavior is mediated by a specific direction in the model's residual stream -- the running sum of information that flows through the network layer by layer. If you prevent the model from representing this direction, it loses its ability to refuse requests.