PropMe Reveals LLMs Leak Far Less Data Than Worst-Case Tests Suggest
A new framework separates what LLMs can be forced to leak from what they actually leak in everyday use, and the gap is surprisingly large.
Takeaways
- New framework: PropMe separates LLM memorization capability (what adversarial attacks can extract) from propensity (what leaks in normal use).
- Big gap found: Prefix attacks elicit 50-token average match lengths vs. ~28 tokens for natural prompts; near-verbatim rates drop from 0.032 to 0.001 in non-adversarial settings.
- SimpleTrace tool: A GPU-free tracing pipeline built on infini-gram that deterministically links model outputs to training corpus text at millisecond speed.
- Continual training reduces leakage: DFM Decoder, fine-tuned from Comma on different data, shows measurably lower memorization propensity for the original training corpus.
- Audit recommendation: The authors argue memorization audits should always report both worst-case extractability and everyday leakage propensity together.
- Limitation: Requires full access to training corpus; closed models with unknown data cannot be evaluated with this approach.
There is a question that has haunted LLM auditors, legal teams, and privacy researchers for years: if a model was trained on your data, will it spit it back out? The answer, it turns out, depends heavily on how you ask. A new paper introduces PropMe, a framework that draws a sharp line between two very different things: what a model can be made to regurgitate under adversarial pressure, and what it tends to reproduce in normal conversation.
The flaw in how we measure memorization today
Large language models can reproduce training data, but existing memorization evaluations mostly measure whether models can be forced to do so, rather than whether they do so under ordinary use. The standard approach, called a prefix attack, works by taking a chunk of text from the training corpus, feeding the model its first 50 tokens as a prompt, and checking whether the model completes it verbatim. This is a worst-case stress test -- useful, but it tells you almost nothing about what happens during a normal user session.
While related work shows that models can reproduce memorized content, it largely evaluates memorization as a capability. Less is known about memorization propensity: whether models tend to reproduce training data under ordinary or weakly targeted conditions. This distinction matters enormously for real-world risk assessment. A model that only leaks data when someone deliberately engineers an adversarial prompt is a very different liability than one that leaks data while answering casual questions.
Capability vs. propensity: a critical distinction
Recent work has argued that LLM evaluations should distinguish between capabilities (behaviors that models can exhibit when successfully elicited) and propensities (behaviors that models tend to exhibit under a given distribution of contexts). PropMe operationalizes this distinction for memorization specifically. The framework introduces a metric transformation that can be applied to any existing memorization metric to produce a propensity-adjusted version of it -- meaning you do not have to throw away your existing tooling, you augment it.