Northwestern Catches LLMs Recycling the Same Code in Evolutionary AI
A new study finds LLMs used as mutation operators in genetic programming reliably converge to the same structural templates, undermining the diversity that evolutionary search depends on.
Takeaways
- Core finding: LLMs used as mutation operators in evolutionary algorithms reliably converge to a small set of structural templates, regardless of model or prompt.
- The numbers: In 87% of mutation chains, over 93% of mutations revisit a previously seen structural skeleton -- only constants change.
- Control comparison: Classical GP subtree mutation does NOT show comparable convergence, confirming the effect is intrinsic to LLMs.
- Reasoning helps: Reasoning-enabled model variants (GPT-5 Mini with reasoning, Gemini with thinking) produce significantly more structural diversity than base models.
- Practical impact: Systems like FunSearch, AlphaEvolve, and QD algorithms using LLM mutation may be exploring far less of program space than assumed.
- Code available: Full experimental code is open-source at github.com/can-gurkan/lmca.
There is a growing wave of systems that use LLMs as mutation engines inside evolutionary algorithms -- from Evolution through Large Models to Google's AlphaEvolve and FunSearch. The pitch is compelling: instead of random bit-flips, you get semantically intelligent edits that understand what a program does. But a new paper from Northwestern University and Augustana College asks a deceptively simple question: when you keep asking an LLM to mutate a program, does it actually explore new territory, or does it just keep rewriting the same thing in slightly different words?
The answer, it turns out, is mostly the latter -- and the numbers are stark enough to matter for anyone building LLM-in-the-loop optimization systems.
The trap at the heart of LLM-driven evolution
Evolutionary algorithms rely on the premise that simple variation operators, applied iteratively, can give rise to a vast and diverse space of candidate solutions. Among these operators, mutation plays a central role, continually injecting new genetic material and enabling populations to explore beyond their current configurations. LLMs seemed like a natural upgrade to this process: they can make changes that preserve semantic intent, respect syntax, and even reason about what a program does. LLMs are well-placed to act as selection and mutation operators, as they have been trained on vast datasets of human knowledge and culture, offering a mechanism through which to make semantically meaningful mutations via text.
But there is a catch. Multiple independent LLM ideation sessions frequently converge around similar themes and concepts. This convergence results in a narrow exploration of the ideation space, and the computer science literature broadly refers to this phenomenon as "mode collapse" -- a fundamental limitation of current LLM design. The new paper, Mutation Without Variation, is the first to rigorously quantify this collapse specifically in the context of program evolution, with a controlled experimental setup designed to isolate the mutation operator itself.