ByteDance Seed researchers report that only 34 of 64 code changes made by LLMs to their own agent harnesses actually generalise across new tasks. The study, HarnessDev, evaluates the code wrapping a model rather than just the answers it produces. An agent harness includes the execution loop, tools, context, state management, recovery mechanisms, and verification steps. Standard benchmarks like Terminal-Bench 2.1 typically keep this wrapper fixed. The new approach flips the target to measure the runnable harness the model writes itself.
2 stages: Creation and Evolution
The Creation phase gives every model the same weak seed. This includes passive file access, search, and process primitives, plus result and trajectory writers. The seed lacks a loop, planner, verifier, retry logic, or stopping rule. Without modification, this baseline scores zero. The model receives a task-family specification, a design tutorial, and one to three development cases. It then builds a full harness, which is frozen before hidden tasks begin.
In the Evolution phase, the model revises its own frozen code using feedback from a fixed set of 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. Each candidate must complete both evaluations as a pair. The budget allows 10 pairs and at most two five-task probes between them. Every official version is later scored on 630 held-out SWE-Pro instances the creator never saw.
Harnesses are graded on capability, defined as task success, and efficiency, measured by executor tokens with creator tokens excluded.
Setup
Six creator LLMs were tested: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro. They worked inside Claude Code 2.1.177, with GPT-5.5 using Codex 0.144.3. Creation spans four domains and five benchmarks totaling 2,207 instances: SWE-bench Pro public split (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46), and BrowseComp (1,266). Each creator builds three harnesses per benchmark, reported as avg@3. Self-Eval runs each harness with its creator, while Unified-Eval runs all with Gemini 3.1 Pro.
Creation results
Under Self-Eval, Opus 4.8 posts the highest average score at 67.8 against a human-engineered reference of 86.2. The gap depends on domain:
- Code: Opus 4.8 reaches 69.3 on SWE-Pro versus the 80.0 reference. Gemini 3.1 Pro leads Terminal-Bench at 68.8 versus 88.8.
- Search: the widest gap. The best BrowseComp score is 52.6 (GPT-5.5) against a 92.2 reference.
- Writing: Opus 4.8 scores 84.6 on EQ-Bench3, above the 83.7 reference.
- ML experimentation: Opus 4.8 (32.9) and Gemini (32.4) beat the 24.0 MLE-bench reference.
The SWE-Pro, Terminal-Bench, and BrowseComp references are external results from OpenAI’s GPT-5.6 report, not re-runs.
Code volume did not predict quality. The 18 code harnesses added 17,111 net lines, yet Gemini added the fewest (1,006) and led Terminal-Bench. Self-test count barely correlated with score, ranging from 0.13 to 0.26, while revision calls reached 0.57.
Much generated machinery is inert. Of 108 code component instances, 72 trigger in real runs and 18 never fire, all of them state and memory. 11 of 18 harnesses define a State class, yet no checkpoint event appears across 26,679 trajectories. 124 of 587 writing features are dead code.
Cost and executor transfer
MLE-bench token use varied roughly 19-fold. GPT-5.5 hit a 19.1 medal rate with 29.3M tokens while DeepSeek V4 hit 19.6 with 208.4M. Swapping the executor to Gemini reshuffled rankings. Qwen gained 17.6 points on BrowseComp and 12.9 on MLE-bench, while Opus 4.8’s SWE-Pro score fell from 69.3 to 33.0. This partly happened because one harness hard-coded a 120-step limit around its original executor. The Opus search harness’s duplicate-query rate jumped from 10.1% to 88.2% after the switch.
Evolution results
Nine lineages, comprising five self-runtime and four fixed-Gemini setups, produced 73 official versions and 64 adjacent switches. All five self-runtime creators improved on held-out tasks, ranging from +1.43 to +4.44 points with a mean of +3.11. Under fixed Gemini, only Opus improved; GPT-5.5 regressed 10.32 points.
Progress was not monotonic. Of 64 switches, 8 regressed on both benchmarks, 16 on one, 27 gained only within the noise band, and 2 showed clear positive evidence. A single commit can vary by about ±4.75 pair-score points. Feedback and held-out scores moved in the same direction only 34 of 64 times (53.1%). Only 2 of 9 declared final versions were held-out optimal. Of 169 new functions or classes, 25 have no caller.
The clearest win: Opus 4.8 noticed 99 of 100 runs reported success while only 48 passed. It traced this to premature completion and added a completion gate. Failure diagnosis was otherwise the weakest step: the dedicated trajectory interface was called only twice.
What it means
Developers should treat self-generated agent code as unstable. A harness built for one environment often breaks when moved to another, or when the underlying model changes. The data shows that most code changes do not translate to real-world improvement. State and memory features frequently sit unused. Builders need to verify that generated logic actually runs before trusting it.
