🔍 Read the full analysis: FrontierCode: The Maintainer’s Test For AI Pull Requests on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, a benchmark that assesses whether maintainers would accept a code change, not just whether it passes tests. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; the results do not establish how models perform across all real-world repositories.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that asks whether a patch meets repository maintainers’ standards, not only whether it passes tests. Claude Opus 5.5 scored highest at 52.2%, but it passed every blocking check in 57.7% of attempts, leaving substantial room for human review.
In results published on October 12, 2026, Artificial Analysis used the benchmark’s 100-task Main set. Claude Opus 5.5 (max with fallback) ranked first, followed by GPT-6.1 Sol at 49.1% and Claude Fable 5.1 at 49.0%. The score is distinct from the pass rate: FrontierCode’s score reflects quality after checks, while the pass rate records attempts that cleared all blocking checks.
That distinction is visible in the leading model’s results. Opus’s 57.7% pass rate means it failed at least one blocking check in the remaining attempts. Nine of the 100 tasks were not solved by any model on any attempt, according to the source report. Results apply to this benchmark setup; they do not by themselves establish performance across every coding task or production repository.
FrontierCode checks patches across six areas, including hidden tests, regression and cleanliness commands, whether an agent’s tests catch the original bug, adaptive tests, limits on files and diff size, and an LLM review against maintainer criteria. Failing any blocking check gives an attempt a zero; passing them all earns a score between 70% and 100%, depending on quality checks. The evaluation also penalizes oversized or out-of-scope changes and tests that do not help verify the fix.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results highlight a gap between producing code that runs and producing a change a repository maintainer can approve. A patch can pass ordinary tests yet still be too broad, poorly tested or unsuitable for the project. FrontierCode is designed to make those review concerns part of the score, so its ranking answers a different question from a benchmark based mainly on test success.
For teams considering AI agents for pull requests, the leading pass rate suggests human review remains part of the process under these conditions. The benchmark does not measure every cost of review in a live team, but it indicates that a high model score should not be read as a guarantee that individual patches are mergeable.
Cost also changes the practical comparison. Artificial Analysis lists GPT-6.1 Sol at $0.81 per task and Opus at $10.15. Opus scored 3.1 percentage points higher, at roughly 12.5 times the listed cost. That is a benchmark-specific trade-off, not a general estimate of what the models will cost in every deployment.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Was Built
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Its version 1.1 followed in July with rules against agents looking up solutions online. According to the source report, more than 20 maintainers created tasks drawn from 36 established repositories, including Celery, Budibase, Uppy and Mattermost, spending more than 40 hours on each task.
The full benchmark contains 150 tasks; Artificial Analysis evaluated the 100-task Main set. The tasks are private, according to the report, to reduce the chance models encountered them during training. Every model was run through the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and up to three attempts per task. GPT-6 Luna (medium) served as the judge for the review component.
The model ranking also differs from general intelligence rankings cited in the report. Claude Sonnet 5.5, second among the compared models on the Intelligence Index, ranked fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. These comparisons show that general benchmark standings do not directly determine performance on this maintainer-oriented coding test.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, quoted in Cognition’s announcement
software maintainers testing software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results cover one benchmark version, one 100-task subset and a defined agent setup. It is not clear from the source report how closely performance on these private tasks predicts performance on the broader mix of issues encountered in active software projects. The report also does not establish whether teams using different tools, prompts or review processes would see the same ranking.
Several models had no FrontierCode result as of October 13, 2026, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published table is not a complete comparison of all models mentioned in the source material. The reasons for those missing results are not specified.
Cost figures are reported per task for the tested configurations; the source does not establish that they represent total costs in every use case. Likewise, a benchmark score is not a direct measurement of how much human review time a model saves. Those questions remain open beyond the leaderboard.
As an affiliate, we earn on qualifying purchases.
Models Awaiting FrontierCode Results
The next comparison to watch is whether models without results are added to the leaderboard and how they perform under the same task set and evaluation conditions. The source report specifically identifies Muse Wonder Full as a model whose future result may draw attention, while offering no score for it as of October 13.
For maintainers and engineering teams, the immediate next step is to treat these figures as one signal when comparing coding agents, alongside testing, review effort, cost and performance on their own repositories. Until broader results or deployment evidence are available, the published findings support a limited conclusion: even the top-scoring model did not clear every blocking check on many attempts.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated code changes against maintainer-oriented checks, including correctness tests, regressions, test quality, change scope and an LLM review. A patch must pass every blocking check to register a passing attempt.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results, with a score of 52.2% and a 57.7% rate of clearing all blocking checks.
Does a 52.2% score mean Opus passed 52.2% of tasks?
No. The reported 52.2% is its FrontierCode score. Its pass rate—the share of attempts that cleared every blocking check—was 57.7%.
Can the results show that AI pull requests are ready to merge without review?
No. The benchmark does not establish that. The leading model failed at least one blocking check in a substantial share of attempts, and the source report says the results cover a specific benchmark setup rather than all production repositories.
Which models were missing from the results?
As of October 13, 2026, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
