Skip to main content
Back

Heretic Slashes Qwen3.5 Refusals from 99% Down to 5%

A 2.8GB distilled reasoning model gets stripped of safety refusals through automated abliteration, running locally on consumer hardware with near-zero refusal rate.

  • New Heretic-abliterated release of DeepSeek-V4-Pro-Qwen3.5-4B drops refusals from 99/100 to 5/100.
  • KL divergence of only 0.0378 from original, preserving reasoning capability almost entirely.
  • Base model scores 91.77% on GSM8K and 76.47% average on MMLU-Pro STEM.
  • Ships as 2.8GB Q4_K_M GGUF, runs locally on llama.cpp, LM Studio, vLLM.
  • Built with Heretic v2.0.0, an Optuna-powered automated directional ablation tool.
  • Apache-2.0 license, with 16K+ downloads already; no coding-specific training included.

The Hugging Face account xtools-at has released a modified Qwen3.5-4B reasoning model that answers substantially more requests in the publisher’s refusal test. The release applies Heretic v2.0.0 to the model’s weights and includes a 2.8GB Q4_K_M GGUF build for local inference. Downloads, configuration details, and larger variants are available on the model page.

The lower refusal rate gives developers a compact checkpoint for local evaluation, red-team tooling, and experiments that require fewer safety-triggered rejections. It also shifts more responsibility to the application: deployments may need their own moderation, access controls, logging, and data-handling safeguards.

A Qwen student with a DeepSeek teacher

The model’s name can obscure its lineage. Qwen3.5-4B supplies the architecture and initial weights, while DeepSeek-V4-Pro supplied teacher responses used during distillation. The base checkpoint was fine-tuned on approximately 250,000 mathematics and STEM samples generated in the teacher’s “Max Effect” mode, following the pipeline used for a related 9B release.

Distillation trains a smaller student model to reproduce patterns in a larger model’s answers. In this case, the process targets mathematical and scientific reasoning within a 4-billion-parameter checkpoint. The subsequent Heretic pass changes refusal behavior without another fine-tuning run.

Heretic edits the refusal direction

Heretic uses directional ablation, informally called “abliteration,” to identify activation patterns associated with refusals. Transformer models carry information between layers through a residual stream. If refusal behavior consistently aligns with a direction in that activation space, the method can reduce the model’s sensitivity to that direction by modifying selected weights.