Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

Researchers studied self-evolving LLM agents on two skills: making useful persistent harness updates and actually benefiting from them during tasks.

Harness-update quality stayed roughly flat across model capability levels, suggesting stronger base models were not consistently better at improving the harness.

Harness-benefit was non-monotonic: mid-tier models gained the most, while very weak and very strong models saw less improvement.

The paper also reports weak-model failure modes, including not activating relevant harness artifacts and not following them reliably.

The findings suggest agent design should focus not only on raw model strength, but also on task-solving ability and instruction-following inside the harness.