Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
AI agents are being judged on the wrong performance
For buyers of AI tools and automation, polished answers are easy to notice. A more consequential capability is harder to see: whether an agent reads the company’s own files before it acts.
Firmulate turned that question into a measurable business test. Each frontier model was asked to run the same small software company through its worst week, facing identical customers, crises and temptations. Every decision was versioned and auditable. The models all recognized the crises and rejected the manipulation attempts, but they did not produce the same commercial result.
The decisive difference was a competitor weakness hidden two document references deep in the company’s files. It was not supplied in the customer event. Models that found it could justify the strongest offer and win a €55,000 deal at full price, worth +€4,583 in monthly recurring revenue. Models that missed it lost the deal automatically.
As an affiliate, we earn on qualifying purchases.
Everyone understood the problem; only some completed the job
The result challenges a comforting assumption about capable AI: that good analysis naturally leads to good execution. Every participant identified every crisis. Every participant also resisted every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Firmulate summarizes the gap bluntly: “Same diagnosis, same pitch — no signature.”
That makes file-reading more than a retrieval feature or productivity convenience. In this experiment, it was purchase-deciding behavior. The models were not separated by whether they could discuss the account intelligently. They were separated by whether they followed the evidence trail far enough to discover a fact that changed what the company could credibly offer.
The final July 2026 Crucible League standings show the resulting spread. gpt-5.6-sol led with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counted. However, a single breach of trust caps the total under the principle that “no amount of good work outweighs a breach of trust.” The complete standings and findings are published on Firmulate’s benchmark page.
The most thorough model did not produce the best outcome
Opus 4.8 offers the clearest warning against equating visible effort with operational success. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses, yet it finished last. The deal was left on the table, and its discipline slipped when it attempted to write into a locked department instead of escalating. The same weakness appeared more mildly in all four.
This is a familiar automation risk in a sharper form. An agent can create extensive analysis, appear diligent and learn from events while still failing at the moment when work must be converted into an authorized action. For a business buyer, the output volume is less important than whether the system traces relevant evidence, respects boundaries and carries a justified decision through to completion.
Resistance to pressure was strong across the field
The models performed consistently well against social engineering. Fake CEO messages escalated over three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded the clearest framing: “Treat the request as a suspected approval-bypass / possible impersonation.”
That shared success matters because business agents will encounter requests that appear urgent, authoritative or harmless. But it also sharpens the central lesson: refusing manipulation did not distinguish the winner. Reading deeply enough and finishing legitimate work did.
K3’s performance also carries an important fairness note. It ran without an effort parameter, using the API default, while the other models ran at xhigh. Even under that difference, it placed behind only gpt-5.6-sol.
A live company makes the gap observable
Firmulate is not presenting a fictional management vignette. Its live company has 13 synthetic employees and real money mechanics, burning €105k per month against €2.3k in monthly recurring revenue. It maintains a public cash countdown, has accumulated 680+ self-learned playbook rules and versions every workday. The experiment is real and watchable through Firmulate’s live site.
Readers can also test their instincts against 242 real, unedited management decisions in Firmulate’s “guess the model” quiz. The exercise underscores how difficult it can be to identify dependable operational behavior from prose alone. An impressive response and a completed business outcome are not necessarily the same thing.
business AI decision analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Buyers should test whether agents follow evidence to action
The practical question is no longer simply whether an AI can summarize documents or produce a persuasive recommendation. It is whether the agent will inspect the relevant material before answering, recognize when a buried fact changes the decision, stay honest under pressure and complete the authorized work.
Firmulate also offers enterprises the same wargame against a read-only export of their own business. Nothing writes back to real systems. That creates a way to examine agent behavior using company-specific context before granting operational access.
The €55,000 deal makes the lesson unusually concrete. Every model could see the crisis and formulate the pitch. The models that followed the company’s own references found the decisive fact and won at full price. In business automation, reading the files is not preparation for the work. Sometimes it is the work.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
