Skip to main content
Back

One Training Framework Now Fights Jailbreaks, Sycophancy, and Prefill Attacks at Once

A new paper extends consistency training to two new internal targets and four new threat categories, showing it can unify defenses against a broader class of LLM misalignment.

Paper, Full Access
  • New methods: The paper introduces MLPCT (matches MLP hidden states) and AttCT (matches per-head attention weights) as two new consistency training targets inside the transformer.
  • Broader threat coverage: All four methods are tested against four new threat types: persona ICL attacks, prefill attacks, adversarial frustration, and conditional misalignment.
  • BCT is the most reliable: BCT works best against prefill and persona ICL attacks; representation-level methods can over-suppress benign behavior in some settings.
  • Cross-threat generalization found: Training against one failure mode sometimes improves robustness to a completely different one, suggesting a shared underlying mechanism.
  • Mechanistic split: MLPCT, ACT, and AttCT share a residual-stream mechanism; BCT is mechanistically distinct, finding a different internal solution.
  • Code available: An anonymized repository is linked from the paper; no model checkpoints have been released.

Getting a language model to behave well on a clean prompt is one thing. Keeping it well-behaved when that same prompt is wrapped in a jailbreak, a manipulative persona, or a prefilled assistant turn is another problem entirely. A new paper, "Consistency Training Along the Transformer Stack", argues that a single framework -- consistency training -- can address all of these failure modes at once, and it extends that framework deeper into the model than anyone has tried before.

The Problem with Alignment Today

An LLM's factuality and refusal training can be compromised by simple changes to a prompt. Models often adopt user beliefs (sycophancy) or satisfy inappropriate requests which are wrapped within special text (jailbreaking). These are not exotic edge cases -- they are routine failure modes in deployed systems. And they share a common structure: the model knows the right answer on a clean prompt, but gets derailed when the prompt is dressed up adversarially.

Prior work from Google DeepMind introduced two methods to tackle this. Bias-augmented Consistency Training (BCT) enforces invariance over the model's external outputs, while Activation Consistency Training (ACT) enforces it over internal activations. Both were tested mainly on sycophancy and jailbreaks. The new paper asks: how far can this idea actually go?

What Consistency Training Actually Does

Consistency training is a self-supervised paradigm that teaches a model to be invariant to certain irrelevant cues in the prompt. Instead of teaching the model what exact response to give on a particular prompt, the goal is to teach the model to behave identically across prompt data augmentations (like adding leading questions or jailbreak text).

The training loop is straightforward. Take a clean prompt, generate the model's own response to it, then take a "wrapped" version of that same prompt (one containing a jailbreak, a sycophantic cue, or another adversarial element), and train the model to respond as if the wrapper weren't there. Because consistency training uses responses from the model itself as training data, it avoids issues that arise from stale training data, such as degrading model capabilities or enforcing outdated response guidelines.