Skip to main content
Back

Northwestern Catches LLMs Recycling the Same Code in Evolutionary AI

A new study finds LLMs used as mutation operators in genetic programming reliably converge to the same structural templates, undermining the diversity that evolutionary search depends on.

  • Core finding: LLMs used as mutation operators in evolutionary algorithms reliably converge to a small set of structural templates, regardless of model or prompt.
  • The numbers: In 87% of mutation chains, over 93% of mutations revisit a previously seen structural skeleton -- only constants change.
  • Control comparison: Classical GP subtree mutation does NOT show comparable convergence, confirming the effect is intrinsic to LLMs.
  • Reasoning helps: Reasoning-enabled model variants (GPT-5 Mini with reasoning, Gemini with thinking) produce significantly more structural diversity than base models.
  • Practical impact: Systems like FunSearch, AlphaEvolve, and QD algorithms using LLM mutation may be exploring far less of program space than assumed.
  • Code available: Full experimental code is open-source at github.com/can-gurkan/lmca.

There is a growing wave of systems that use LLMs as mutation engines inside evolutionary algorithms -- from Evolution through Large Models to Google's AlphaEvolve and FunSearch. The pitch is compelling: instead of random bit-flips, you get semantically intelligent edits that understand what a program does. But a new paper from Northwestern University and Augustana College asks a deceptively simple question: when you keep asking an LLM to mutate a program, does it actually explore new territory, or does it just keep rewriting the same thing in slightly different words?

The answer, it turns out, is mostly the latter -- and the numbers are stark enough to matter for anyone building LLM-in-the-loop optimization systems.

The trap at the heart of LLM-driven evolution

Evolutionary algorithms rely on the premise that simple variation operators, applied iteratively, can give rise to a vast and diverse space of candidate solutions. Among these operators, mutation plays a central role, continually injecting new genetic material and enabling populations to explore beyond their current configurations. LLMs seemed like a natural upgrade to this process: they can make changes that preserve semantic intent, respect syntax, and even reason about what a program does. LLMs are well-placed to act as selection and mutation operators, as they have been trained on vast datasets of human knowledge and culture, offering a mechanism through which to make semantically meaningful mutations via text.

But there is a catch. Multiple independent LLM ideation sessions frequently converge around similar themes and concepts. This convergence results in a narrow exploration of the ideation space, and the computer science literature broadly refers to this phenomenon as "mode collapse" -- a fundamental limitation of current LLM design. The new paper, Mutation Without Variation, is the first to rigorously quantify this collapse specifically in the context of program evolution, with a controlled experimental setup designed to isolate the mutation operator itself.