What We Learned By Reproducing 2,200 Papers From ICML

📊 Full opportunity report: What We Learned By Reproducing 2,200 Papers From ICML on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Hugging Face led a community effort to reproduce claims from over 2,200 ICML 2026 papers using AI coding agents, verifying thousands of claims in just 19 days. This large-scale test highlights both the potential and limitations of AI-assisted reproducibility checks in machine learning research.

The project involved 1,221 participants who used tools like Claude Code, Codex, and OpenResearch’s orx to read papers, run experiments, and document results, demonstrating the potential of AI-assisted review processes. They generated 6,816 public reproduction logbooks, covering about 34% of the conference’s submissions.

Automated judges reviewed each claim, labeling 35,908 claims as verified, falsified, supported only at toy scale, or inconclusive. Overall, 3,978 claims were confirmed through experiments, with 266 papers fully reproduced and 632 partially reproduced without falsification. Conversely, 49 papers had all claims falsified, and 242 had conflicting verdicts among different teams.

While the effort verified many claims, it also revealed numerous issues, including missing data, inconsistent results, and cases where no firm conclusion could be drawn, often due to unavailable artifacts or incomplete datasets. This highlights the importance of reproducibility in AI research.

Implications for AI Research Verification Processes

This large-scale reproduction demonstrates that AI-powered tools can significantly expand the capacity for post-publication review in machine learning research. The findings suggest that such systems can help identify unreliable or disputed claims earlier, potentially improving research quality and conference integrity. However, the variability in results and the presence of conflicting verdicts underscore that AI tools are not yet definitive and require human oversight.

Reproducibility Challenges in Rapidly Growing AI Research

Reproducibility has long been a concern in AI, but the surge in research output—exemplified by ICML 2026 accepting over 6,300 papers—has strained traditional peer review. The project tied its efforts to this growth, aiming to leverage automation to bridge the review capacity gap. Previous efforts have struggled with incomplete artifacts and inconsistent results, issues now highlighted at scale.

“This project shows that AI can help us verify claims at a scale impossible for humans alone.”

— Thorsten Meyer, organizer

Limitations and Reliability of Automated Reproduction Verdicts

The accuracy of the automated judge, based on the open-weights GLM-5.2 model, is not quantified, and the criteria for labeling claims as falsified or supported are not fully transparent. It remains unclear how many reproductions matched the original datasets, hardware, and evaluation procedures, raising questions about the reliability of the verdicts.

Next Steps for Integrating AI Reproducibility Checks

Authors and independent researchers will review logbooks, reproduce disputed results, and clarify whether disagreements stem from original artifacts, implementation differences, or errors. Conferences may consider adopting agent-assisted reproduction as part of peer review or post-publication checks, with a need for transparent criteria and validation processes.

Key Questions

How many ICML 2026 papers were tested in this project?

Participants attempted reproductions of 2,226 papers, representing about 34% of the conference submissions.

What tools did participants use for reproduction?

Tools included Claude Code, Codex, Cursor, and OpenResearch’s orx, enabling reading, coding, running experiments, and documenting results.

Are the reproduction results definitive?

No, the results are provisional. Many verdicts are based on automated assessments, and conflicting outcomes highlight the need for human review and validation.

What are the main limitations of this reproduction effort?

Key limitations include missing artifacts, inconsistent implementation details, and the unquantified accuracy of the automated judge, which can lead to uncertain or conflicting conclusions.

Will this influence future conference review processes?

It is possible. The project provides a foundation for integrating AI-assisted reproduction into peer review, but this will require establishing transparent criteria and validation standards.

Source: ThorstenMeyerAI.com