MAEM Generates Text to Expose Hidden Directions Inside Any AI Model
A new interpretability method trains a language model to generate text that provokes any target direction inside another model, verifiable by forward pass.
Takeaways
- New method trains a language model (MAEM) to generate text activating any target direction in another model.
- Uses a label-free RL objective maximising cosine similarity between target direction and subject activations.
- A Qwen3.6-27B MAEM activates 89% of held-out SAE features with 8 samples.
- Beats 10M-token corpus search on a majority of held-out languages and domains.
- Recovers LoRA backdoor payloads and verbalises model inner monologue during shutdown scenarios.
- Unlike natural-language autoencoders, exemplars are verifiable by running them through the subject model.
Mechanistic interpretability often identifies a language model’s internal concepts by searching a large corpus for passages that strongly activate a chosen direction. The method works for common concepts represented in the corpus, but its coverage drops for rare features, specialized domains, underrepresented languages, and directions that natural web text seldom triggers.
A new paper, Towards Interpreting Any Residual Stream Direction by Scaling Activation Inversion, proposes generating the activating text. Its Maximally-Activating Example Metamodel, or MAEM, accepts a residual-stream direction and produces text whose activations align with it. An HTML version includes the experiments and figures.
Why Corpus Search Hits a Ceiling
A transformer’s residual stream is the running vector of activations passed through its layers, carrying information that later computations use to produce the next token. Interpretability researchers often represent a concept or behavior as a direction in that vector space. Text associated with the concept should move the model’s activations toward the corresponding direction.
Corpus search ranks passages by how strongly they activate that target. Its accuracy therefore depends on coverage: the relevant text must already exist in the dataset. Larger corpora improve the odds, but each expansion adds storage and inference costs while still missing concepts absent from realistic web crawls.
A Generator Replaces the Index
MAEM learns to write candidate passages that activate a supplied direction. During reinforcement learning, the subject model processes each generated passage and returns a reward based on cosine similarity, a measure of directional alignment that ignores vector magnitude. Higher similarity means the resulting activation points more closely toward the target.