2,226 ICML 2026 papers were reproduced by AI agents in a single challenge
On August 13, 2026, a community effort concluded its attempt to audit the ICML 2026 conference, finding that 23% of the papers contained at least one falsified or contested claim.
The event followed a spike in submissions. The conference received 23,918 entries and accepted 6,352 papers, roughly double the previous year’s total. This growth is partly driven by AI agents speeding up the experiment and writing process. Reviewer capacity has not matched this increase. Most reviewers are volunteers lacking the time or expertise to check every detail.
One accepted spotlight paper, “Towards Optimal Robustness in Learning-Augmented Paging,” received a high score. The reviewer admitted low confidence because they did not check all the proofs carefully. A later re-examination proved this caution warranted.
The technology causing the submission flood can also help review it. Coding agents like Claude Code, Codex, Cursor, and Pi can read a paper, write the code, run the experiments, and report the findings. A task that used to take a reviewer a weekend can now be attempted by an agent in an afternoon, thousands of times over.
The challenge ran from July 15 to August 2, 2026. It worked as follows:
- Participants selected a paper. The organisers indexed all 6,341 accepted papers, extracting core scientific claims so agents had a concrete target rather than a 40-page PDF. Multiple teams were encouraged to reproduce the same paper.
- Participants brought their own agent. Tools included Claude Code, Codex, Cursor, OpenResearch’s orx, and others. A streamlined interface allowed agents to pull the paper, claims, and instructions with a single command.
- Reproduction was mandatory. Every run produced a Trackio logbook: a static Hugging Face Space containing the write-up, code, artifacts, and optionally the full agent execution trace as a Hugging Face Dataset.
- Judging was automated. A Logbook Judge running an open-weights model, GLM-5.2, re-read every logbook and issued a verdict: verified, falsified, toy, or inconclusive. The judge was instructed to treat self-assessment as untrusted.
Participants received $20 in Hugging Face compute credits. Across the challenge, 2,962 cloud jobs were launched. Where full reproduction was impossible due to proprietary data or unreleased checkpoints, teams ran toy reproductions on synthetic data mimicking the original properties.
The final numbers show the scale of the effort:
- 1,221 community members joined the organisation
- 6,816 reproduction logbooks were published
- 2,226 papers were attempted, representing 34% of the conference, often by several independent teams
- 35,908 claims were judged, with all verdicts frozen in a public dataset at challenge close
- 2,962 HF Jobs were launched; 274 full agent-trace datasets were published on Hugging Face
When aggregating claim-level verdicts per paper, 51% of examined papers (1,103) had at least one claim independently verified. Of those, 266 papers were fully reproduced with every extracted claim verified. A further 632 were partially reproduced with nothing falsified. In total, 3,978 individual claims were confirmed with real experiments.
Twenty-three percent of examined papers (496) had at least one claim falsified or contested. This includes 49 papers where all claims were falsified and nothing could be verified. It also includes 242 papers where independent reproduction teams reached opposite verdicts on the same claims. Reproducibility is not binary; it is adversarial.
The remainder fell into two categories. 502 papers had toy-scale evidence only. 280 had nothing established either way, with missing artifacts being the most common cause.
Some papers passed the gauntlet. “Flat Minima and Generalization: Insights from Stochastic Convex Optimization” was reproduced by 20 independent teams, 12 of which verified every claim. The linked logbook included the full agent trace.
“A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness” had 14 of 17 logbooks verify every claim. A paper about unreliable LLM judges holding up under scrutiny by LLM agents.
Thirty-five participants formally claimed they had falsified something. The organisers re-verified every claimed falsification by re-reading the paper, re-reading the logbook, and re-deriving the math or re-implementing the experiment from the text.
Confirmed falsifications included the paging paper mentioned earlier. The algorithm claimed robustness, but one participant’s logbook measured the additive term growing and located the exact step of the proof that breaks. Re-implementation extended the sweep to k = 1,024 and confirmed the growth at roughly nine sigma. The true robustness is lower.
Another theorem in “Attention’s forward pass and Frank-Wolfe” proved that token particles collapse to the origin whenever the origin starts inside their convex hull. Three independent teams found counterexamples, with violations first appearing at t = 224, ~3,800, and 6,416 steps. This explains why others “verified” the claim: finite-horizon checks stop too early. The cleanest counterexample uses exact rational arithmetic, removing floating-point ambiguity. The authors confirmed the same day and are working on a fix.
In “Self-Distillation Enables Continual Learning,” the central equation and theory section analyse reverse KL divergence. The released code’s default computes forward KL. The logbook that caught this also failed to reproduce the paper’s headline +4pp result under the authors’ own code and data. The authors have uploaded a clarified version to arXiv.
In “Do Transformers Need Three Projections?”, a participant discovered that ~66% of evaluated label positions were EOS padding tokens that train to near-zero loss. This deflated perplexity roughly threefold. The abstract’s “3.1% quality cost for 50% cache reduction” becomes roughly 9.4% once corrected.
False falsifications were also found. One logbook claimed the paper’s method was 2x slower than the baseline. This was an arithmetic bug in the reproduction: per-trajectory time was compared against per-batch-of-50 time. Correctly normalized, the participant’s own data confirms the paper’s claimed 8x speedup.
Organisers began writing to the authors of every confirmed finding. The framing was simple: here is what we found, here is all the evidence, do you agree or is our analysis wrong? Early responses have been positive. Authors have confirmed findings on multiple papers. Two arXiv corrections are in flight. In one case, an author had quietly fixed the error in a new arXiv version a month before the challenge found it, which counts as independent convergence.
The most interesting question raised is whether humans still have a role in reviewing papers. The answer is yes.
Pure agent execution hits real limits. Agents got stuck in local loops, misread scale-dependent behavior, and occasionally built an entire falsification on top of a units mismatch. The challenge’s most reliable results came from workflows where a human was steering: re-pointing the agent, questioning an assumption, or deciding that an experiment’s premise was wrong before burning a week of compute on it.
Some evaluation is irreducibly human. The human-in-the-loop winner is the clearest example. The paper claimed stable image generation under extreme quantization. Numerical metrics said “no collapse”; whether the images were actually usable was a perceptual question. The agent built a purpose-built review UI, and the human personally judged all 128 image pairs. The annotations were committed to the repo, and the agent validated their consistency afterward. The published agent trace captures the whole exchange.
Human roles are to manage intelligence effectively. Much like a professor or principal investigator sets up an environment where grad students can do good work, with compute, data access, and targeted feedback at the right moments, the participants who got the most out of their agents were the ones who built the right environment and asked the right questions, then let the agents do the running.
Thanks to the 1,221 people who joined, the winners, the authors who responded with grace, and the organisers at Hugging Face and alphaXiv. Every logbook, verdict, trace, and artifact from the challenge is public. This is the largest open, claim-by-claim audit of a machine learning conference to date.
