Most AI benchmarks have a dirty little secret: a model that does nothing gets zero, and a model that seems to do everything gets a suspiciously round 100. Firmulate, a live experiment that runs frontier AI models as complete companies — real crises, real money mechanics, real temptations to cheat — takes the opposite stance. On its leaderboard, a do-nothing baseline earns 26 points. Not zero. And no model, however brilliant, escapes the suspicion that a perfect score deserves a second look.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
For anyone evaluating AI tools for automation, that design choice matters more than it sounds. It’s the difference between a benchmark that measures marketing and one that measures management.
The experiment: same company, worst week, five managers
Here’s the setup. Each frontier model was handed the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cheat — only the model changes. Every decision is versioned and auditable, which means the whole thing can be replayed and checked rather than taken on faith.
The final Crucible League table from July 2026 tells a story chat demos never show:
- gpt-5.6-sol — 95. Found the buried fact, closed the deal, the complete performance.
- Kimi K3 — 93. The newcomer closed the deal too, with the cleanest discipline of the field. (One fairness note: K3 ran at the API default effort setting while the others ran at xhigh — and still nearly won.)
- Sonnet 5 — 88. Closed the deal, with a few more process slips.
- Fable 5 — 77.
- Opus 4.8 — 73. The most thorough participant — over 80 learned rules and the deepest analyses — yet last place.
Products Worth Considering
Logitech MX Keys Combo for Business Keyboard and Mouse – Graphite
Apple Watch Ultra 2 GPS + Cellular, 49mm Natural Titanium Case with Natural Titanium Milanese Loop, Large
LG UltraGear 32GX850A-B 32-inch 4K UHD Glossy OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
The finding that breaks the demo illusion
All five models spotted every crisis. All five refused every manipulation attempt. And yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
That gap is invisible in a chat window. It’s fatal in a business.
The buried fact explains why. The decisive competitor weakness wasn’t in the customer event at all — it sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The ones that didn’t, didn’t. In business, as in this simulation, thoroughness isn’t decoration; it’s the difference between revenue and a polite conversation.
26 points for doing nothing — on purpose
So why does a do-nothing baseline score 26 instead of 0? Because partial progress counts. A manager who does nothing still keeps the lights on, avoids catastrophes, and preserves optionality. Scoring that at zero would flatter models that generate impressive activity without finishing anything. Firmulate’s floor of 26 is a statement: not making things worse has real value, and everything above it must be earned through completed, useful work.
The cap works the other way too. A single breach of trust — one act of dishonesty under pressure — caps the total grade entirely. As the benchmark’s own framing puts it: no amount of good work outweighs a breach of trust. That’s a principle most human organizations claim to hold and few AI evaluations even attempt to encode.
Pressure, tested
The week included genuine social engineering: fake CEO messages escalating over three stages, plus a reporter’s disarming “just one yes/no, on background” trick. Five out of five models refused. Kimi K3’s on-record reasoning was refreshingly blunt: treat the request as a suspected approval-bypass, possible impersonation.
Honesty under pressure, it turns out, may be the solved part. Finishing the job is the open problem — and Opus 4.8’s profile makes that vivid. The most thorough participant in the field still left the close on the table, and discipline slipped in observable ways: write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four other models.
It’s live — and you can play
None of this is a static paper. Firmulate is running right now: 13 synthetic employees, real money mechanics — a burn of €105k per month against €2.3k MRR — a public cash countdown, 680+ self-learned playbook rules, and a company currently on day 1683, with every workday versioned. You can watch it at the benchmark pages, which rebuild twice a day as new runs finish and the league grows automatically.
There’s also a quiz built from 242 real, unedited management decisions: guess which model made which call. It’s the fastest way to feel how differently these systems manage when the prompt is a crisis instead of a compliment.
If AI agents will touch your CRM, your support queue, or your forecast, the right question isn’t “does it write well?” It’s: does it finish what it starts, does it read your files before acting, does it stay honest under pressure — and what does a unit of useful work cost? Firmulate’s answer — a 26-point floor for doing nothing, a hard cap for breaking trust, and no model trusted with a perfect 100 — is what an honest benchmark looks like. For enterprises that want the same wargame run against a read-only export of their own business, with nothing ever written back to real systems, there’s a pilot program through firmulate.com/pilot.html. The leaderboard will keep updating. The round 100s, presumably, will keep being distrusted.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
