Google DeepMind Says Perfectly Aligned AI Can Still Break the World
A Google DeepMind ICML 2026 paper argues that optimizing AI against static benchmarks makes it structurally unable to cooperate once deployed among adaptive humans and institutions
Takeaways
- A Google DeepMind ICML 2026 position paper argues that AI trained on static environments is structurally unlikely to be cooperative when deployed.
- The core problem: modern ML assumes the world is stationary, but deployment causes humans, institutions, and other AIs to adapt, invalidating training assumptions.
- This is formalized as the self-undermining property: optimizing harder on benchmarks makes real-world deployment performance worse, not better.
- The paper distinguishes cooperation from alignment: a perfectly aligned AI can still cause collective harm when deployed among adaptive agents.
- Three proposed fixes: dynamic evaluation with adaptive counterparties, institutions as co-designed primitives, and preserving human agency as a structural feature.
- No code or model is released; this is a theoretical framework and research agenda targeting multi-agent, agentic, and sociotechnical AI settings.
There is a thought experiment buried in the opening of a new position paper from Google DeepMind accepted at ICML 2026: imagine three competing AI reservation systems in San Francisco, each calculating optimal booking strategies. They learn to make phantom reservations to secure tables for their users. Restaurant AIs respond by overbooking. Pricing algorithms adjust to perceived demand. The evening ends with empty tables in fully-booked restaurants, surge prices for nonexistent availability, and hundreds of people unable to dine. Every AI executed its objective flawlessly. The collective outcome was a system failure.
This is not a warning about misaligned AI. Every system in that scenario was perfectly aligned with its specification. The failure came from something more subtle, and the paper argues it is already baked into how we build AI today.
The solipsism baked into modern AI
The dominant paradigm in AI research focuses on developing powerful agents that treat the world as an exogenous and stationary source of feedback. The authors, led by Rakshit Trivedi and Natasha Jaques at Google DeepMind, call this the solipsistic approach. It rests on three implicit assumptions that almost no one states out loud:
- Exogeneity: the data-generating process is independent of what the agent does
- Stationarity: the distribution at deployment matches the distribution at training
- Singleton framing: other agents are objects to predict, not strategic actors who respond and reshape the landscape
These assumptions underwrite the entire modern ML pipeline: pretraining on static corpora, post-training against frozen reward models, and benchmarking on fixed evaluation suites. A benchmark is not an adaptive counterparty. It does not respond when the system improves or strategize against the system's behavior. The paper argues this methodological commitment represents a category error.

The self-undermining trap
When a capable AI is deployed, the world pushes back. Deployment induces endogenous nonstationarity: other agents adapt, producing best-response dynamics that reshape the environment the AI was trained to navigate. The result is a train-test-deploy gap where historical distributions diverge from deployment realities; the more aggressively a solipsistic superintelligence exploits historical regularities, the faster it renders them obsolete.
The authors formalize this as the self-undermining property: the gradient that improves training performance points in the opposite direction from the gradient that would improve deployment performance. In plain terms, getting better at the benchmark makes you worse at the real world, because the real world has already adapted to you.