Heretic Slashes Qwen3.5 Refusals from 99% Down to 5%
A 2.8GB distilled reasoning model gets stripped of safety refusals through automated abliteration, running locally on consumer hardware with near-zero refusal rate.
Takeaways
- New Heretic-abliterated release of DeepSeek-V4-Pro-Qwen3.5-4B drops refusals from 99/100 to 5/100.
- KL divergence of only 0.0378 from original, preserving reasoning capability almost entirely.
- Base model scores 91.77% on GSM8K and 76.47% average on MMLU-Pro STEM.
- Ships as 2.8GB Q4_K_M GGUF, runs locally on llama.cpp, LM Studio, vLLM.
- Built with Heretic v2.0.0, an Optuna-powered automated directional ablation tool.
- Apache-2.0 license, with 16K+ downloads already; no coding-specific training included.
The Hugging Face account xtools-at has released a modified Qwen3.5-4B reasoning model that answers substantially more requests in the publisher’s refusal test. The release applies Heretic v2.0.0 to the model’s weights and includes a 2.8GB Q4_K_M GGUF build for local inference. Downloads, configuration details, and larger variants are available on the model page.
The lower refusal rate gives developers a compact checkpoint for local evaluation, red-team tooling, and experiments that require fewer safety-triggered rejections. It also shifts more responsibility to the application: deployments may need their own moderation, access controls, logging, and data-handling safeguards.
A Qwen student with a DeepSeek teacher
The model’s name can obscure its lineage. Qwen3.5-4B supplies the architecture and initial weights, while DeepSeek-V4-Pro supplied teacher responses used during distillation. The base checkpoint was fine-tuned on approximately 250,000 mathematics and STEM samples generated in the teacher’s “Max Effect” mode, following the pipeline used for a related 9B release.
Distillation trains a smaller student model to reproduce patterns in a larger model’s answers. In this case, the process targets mathematical and scientific reasoning within a 4-billion-parameter checkpoint. The subsequent Heretic pass changes refusal behavior without another fine-tuning run.
Heretic edits the refusal direction
Heretic uses directional ablation, informally called “abliteration,” to identify activation patterns associated with refusals. Transformer models carry information between layers through a residual stream. If refusal behavior consistently aligns with a direction in that activation space, the method can reduce the model’s sensitivity to that direction by modifying selected weights.