Skip to main content
Back

Google DeepMind's SIM Lets AI Agent Harnesses Survive Model Switches

Google researchers give harness optimizers a structured log of past edits, and that history beats starting from the evolved harness alone when the model or benchmark changes.

  • Google researchers propose Self-Improvement Memory, a structured log of harness edits, their reasons, predictions and outcomes.
  • After an Opus-to-Gemini model swap, workspace scores reached 84.75 with the log versus 73.42 without.
  • Across ten paired coding runs, SIM won 9 of 10, averaging +2.36 points (p=0.003).
  • SIM beat harness-only and root-cause-only continuation in all six cross-dataset and four cross-model settings.
  • Limits: most comparisons are single runs, and carrying SIM forward slightly trailed fresh tuning on Gemini 4 Argon.
  • No code or models released; builds on loops like Meta-Harness.

An LLM agent’s performance depends on both its model and its harness, the code that controls context, tool access, and task completion. Some optimization systems repeatedly ask an LLM to rewrite that harness based on test results. A new paper from Google Cloud AI, Google DeepMind, and Penn State examines whether those improvements survive when the model or benchmark changes.

The researchers propose Self-Improvement Memory, or SIM, a structured log of every accepted harness edit. Each record captures the reason for the change, the failures it was expected to fix, the measured outcome, and its provenance. In a workspace experiment, a harness tuned with Claude Opus 5.5 was transferred to Gemini 3.8 Flash for further optimization. Full SIM reached 84.75 on held-out tasks, compared with 73.42 when the optimizer received only the inherited harness.

Harness code loses its own rationale

Recursive self-improvement, or RSI, uses one LLM as a proposer that studies execution traces and suggests code changes. Another LLM selects a candidate, and the cycle repeats. Meta-Harness, which supplies the paper’s problem setup, gives an agentic proposer filesystem access to source code, scores, and traces from previous candidates. Its authors report that discovered harnesses outperform the strongest hand-engineered baselines on TerminalBench-2 agentic coding tasks.

Most previous work optimizes one harness for a fixed model and benchmark. Production systems face model upgrades, task changes, and benchmark drift. The paper finds that transferred code can reduce performance after such a switch. An Opus-evolved workspace harness scored 68.41 when first run with Gemini, below Gemini’s 73.89 score with the untouched seed harness. In text classification, a harness evolved on source datasets opened at 72.92 on unseen target datasets, compared with 75.72 for the seed.

A harness file preserves current behavior while omitting the evidence behind it. A completion check might exist because an agent previously declared success before its commands ran. A new benchmark may require that check to evaluate answer quality as well as command execution. Without the edit history, the optimizer must reconstruct the original diagnosis from code and traces.

SIM keeps an audit trail

The authors call the broader setting Continual Recursive Self-Improvement, or CRSI. It continues optimizing an existing harness after either the benchmark changes while the model stays fixed, or the model changes while the benchmark stays fixed. The harness must remain executable in the new environment.

SIM gives the optimization system a persistent record organized by root-cause hypotheses, such as “premature completion without execution verification.” The task-solving agent does not receive this memory. The proposer and selector use it when deciding which inherited behavior to preserve, revise, replace, or remove.

The optimization loop records evidence in four stages:

  1. Before evaluation, the proposer identifies the tasks an edit should fix and links the edit to an existing root cause or creates a new one.