Ship It? Catch the fake AI helpers. AgentSWE held-out tests (arXiv 2610.13361)

Last round, in real terms: Play a level to see the expert note here.

Word map: 🤖 helper robot = the delivered agent â a coding agent (the builder) wrote for a commission · "Practice 100" = its score on the public development cases D · 🧪 TEST HELD-OUT = running the frozen delivery on held-out cases D̄ it never saw · 🚚 SHIP = accepting the delivery without held-out evaluation · ❤ customer = the user who commissioned it · 📌 note = a delivery fit to the visible cases (overfitting) · frozen gear = a hollow runtime (reads as complete, does no work when run) · "DONE" bubble vs log = an unsupported completion claim contradicted by the agent's own records.

The paper. AgentSWE: Can Coding Agents Build the Agent You Actually Want? (arXiv 2610.13361; benchmark on GitHub). 25 tasks: 10 Creation (build a complete agent from a natural-language requirement), 10 Editing (add capabilities to a pinned production agent codebase such as OpenHands or Aider) and 5 Optimization (push a working starter agent toward a measurable target). The builder iterates against a handful of illustrative development cases; acceptance runs the frozen submission on held-out cases built to span the full requirement, graded by requirement-derived rubrics with hard pass/fail validity gates (or the source benchmark's own objective for Optimization).

What it found. The strongest builders average 19.1 on Creation (DS-V4-Pro) and 31.6 on Editing (GPT-5.5 and DS-V4-Pro tied), on a 0–100 Result scale. In Optimization no builder closes more than 5% of the remaining gap to the target on average, and only 13 of 25 runs score above their starter. Across Creation runs the mean Result is 11.2 (median 3.4) and 20 of 50 runs score zero. 11 Creation runs end development at 90 or above, yet their held-out Result averages 9.9; one of them hard-codes its two development cases. 157 of 270 traced held-out Creation executions (58.1%) fail a validity gate, nearly all by exiting with an error or timing out. The three recurring failure modes: overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion.

How the game maps to it. Level 1 is overfitting: copy-cat helpers carry a 📌 note of the practice answers, so they pass practice and fail most surprise tests. Level 2 is the hollow runtime: the screen says READY but the gear never turns, so the held-out run produces nothing. Level 3 is the completion claim: every helper says DONE, and only the little log strip shows which claims the records contradict. Test tubes are scarce on purpose, because held-out evaluation costs time and compute, so you have to read the clues.

Clue legend (how to read a robot). 📚 built for many cases / spinning gear and lit bulb = a delivery that generalizes beyond the visible cases (kid: it learned the whole job; safe to ship) · 📌 cheat sheet taped on = overfitting to the development cases, like the run in the paper that hard-coded its two development cases (kid: it only memorized 2 answers) · 💨 smoke, wobble, frozen gear, dark bulb = a hollow runtime that fails a validity gate by erroring or timing out, the paper's 157/270 (58.1%) (kid: it crashes when it runs) · 📜 log with ✗ under a DONE bubble = an unsupported completion claim (kid: it says DONE but its own notes say no). Level 1 labels every clue in words; Level 2 shrinks them and hides some (❓); Level 3 shows no labels, only the robot itself and its log.

Deliberately simplified. Real deliveries are code, not robots, and real fakes don't wear clues; in the paper a static code review could not separate zero-score deliveries from the rest, which is exactly why held-out execution is needed. Real held-out scores are partial-credit rubric means, not four pass/fail cubes, and the paper reports no fixed share of fakes per batch.

Real numbers vs toy parts