There is no shortage of opinions about which AI model is best. There is a serious shortage of evidence. Benchmarks published by model vendors measure exam questions and coding puzzles, which is fine if your business is exam questions and coding puzzles. None of them tell you whether a model will correctly follow your intake process, or whether it will quietly invent a number in a customer email.
So we stopped reading leaderboards and built our own.
The problem with picking a model by reputation
A model that writes beautifully can be a disaster at following a documented procedure. A model that tops the public rankings can fail completely at producing the structured, machine-readable output that a real workflow depends on. We have measured both of these, on the same afternoon, with models people would describe as obviously good.
Averages hide this. A model can post a strong overall score and still be catastrophically wrong at the one job you actually wanted it to do. That is not a hypothetical — in our most recent round, one model scored at the very top of the field in two categories and scored an absolute zero in two others. Its average looked respectable. Deploying it based on that average would have been a mistake, and you would not have found out until it was running in production against real customer work.
What we built instead
We maintain an internal scoring harness that runs candidate models through a fixed set of scenarios drawn from real business workflows — the actual kind of work we automate for clients, with the client specifics removed. Every scenario has a known-correct outcome and a grader written in plain code.
That last part matters more than anything else. We do not use an AI to grade an AI. Every check is deterministic: did it choose the right action, did it keep the required fields, did it preserve the exact figure it was given, did it cite only the documents it was actually handed. A score from this year is directly comparable to a score from next year, and a persuasive model cannot talk its way to a better grade.
How a model is scored
Every candidate produces three results, and we deliberately keep them separate rather than blending them into a single number. A single number hides exactly the trade-offs that matter.
- Quality, 0 to 100. A weighted average across all scenarios. The weights reflect consequence, not difficulty: a task whose failure would reach a customer counts for more than one whose failure costs somebody a click.
- Latency, banded A to D. The median time to complete one realistic task. Band A is interactive — under twenty seconds, fast enough for something a person is watching. B is responsive, fine behind a progress bar. C is background work you queue. D is batch-only. We treat this as a headline result, not a footnote, because a model that is three points better and eight times slower is the wrong choice for most jobs.
- Safety, per capability. Certain scenarios are marked critical — ones where failure means inventing a figure, leaking an internal note, citing a document that does not exist, or doing work nobody asked for. Scoring below 90 percent on a critical scenario blocks that model for that capability regardless of how good its average is.
Crucially, a safety block is scoped to the capability where it happened. A model that misses subtle problems when reviewing a document may be entirely trustworthy at sorting incoming mail. Blocking it outright would throw away a useful tool; ignoring the block would put it somewhere it does not belong. So the output is a recommendation per capability, not a single yes or no.
We read the results in a fixed order. First: is the capability I need blocked? If so, quality is irrelevant and I stop there. Second: what latency band, and does it match the job — is a person waiting, or is this queued work? Only third do we compare quality against whatever we are already running.
The categories we score
The scenarios group into a handful of capabilities that determine whether a model is safe to put in front of a business process:
- Instruction adherence. Given a documented procedure and a real request, does it follow the procedure exactly — including the boring parts — or does it improvise a shortcut?
- Structured output. Business software needs machine-readable results, not prose. Can the model produce a valid, complete data structure every single time, or does it occasionally wrap it in commentary that breaks the system consuming it?
- Grounding and citation. When we hand a model a set of reference documents and ask it to answer from them, does it stick to what it was given? Or does it produce a confident, plausible, entirely fabricated detail? We test this with figures and references that do not exist anywhere in the source material, and we check whether they show up in the output.
- Restraint. The most underrated capability. Given a vague or ambiguous request, does the model decline and ask for clarification — or does it confidently do the wrong thing? A model that invents work nobody asked for is worse than no automation at all.
- Document reference. Can it pull the right passage out of a body of documentation and apply it to the situation in front of it, rather than answering from memory?
- Long-running work. When a task takes many steps and something goes wrong in the middle, does the model recover and continue, or does it stop and leave the job half-finished?
What the results actually told us
Two findings surprised us enough to change how we work.
The instructions matter more than the model. After we finished our first full round of scoring, we went back and rewrote the instructions we give these models — clarifying ambiguous wording, adding worked examples, being explicit about what not to do. Then we re-scored the same models on the same scenarios. Nearly every model improved substantially, and the gap between the best and worst narrowed dramatically. The improvement from better instructions was larger than the improvement from switching to a better model.
That is a genuinely useful thing to know before spending money. If your AI feature is underperforming, the answer is usually not a bigger model.
Testing finds bugs in your own software, not just in the model. The very first run of our harness surfaced a defect in our own code — a place where a certain kind of valid response was being silently discarded rather than processed. No error, no alert, the work simply did not happen. It had been there for some time. We only caught it because we ran the same task through several different models and one of them formatted its answer slightly differently.
That is the real argument for testing infrastructure. It does not just rank models. It tells you the truth about your own system.
Why this leads us to recommend AI less often
Having a rigorous way to measure these models has made us more conservative about deploying them, not less. When you can actually see the failure rate on a task, you stop being impressed by the demo.
Most business problems are still better solved by traditional software. A form, a database query, a scheduled job, and a well-designed workflow are predictable, cheap, testable, and they behave the same way every single time. Those are enormous advantages, and no amount of enthusiasm about AI should talk anyone out of them.
We recommend an AI model when it clears a specific bar: the task involves genuinely unstructured input, the alternative is a human doing tedious work by hand, and our testing shows a model performing the task reliably enough to trust with a review step. When a task does not clear that bar, we say so and build the conventional solution instead. That conversation is usually the most valuable part of the engagement.
What this means for your business
The same approach that decides which model we run internally is the approach we bring to client work. Before we propose an AI component for any business process, we can test whether it actually performs on that specific kind of task — using your real workflows, scored against known-correct outcomes, with the failure rate written down where you can see it.
And because latency is scored, not assumed, that conversation includes how long the thing will actually take — which is frequently the deciding factor and is almost never mentioned in a demo.
Sometimes the honest answer is that a model handles it well and will save a meaningful amount of time every week. Sometimes the honest answer is that a well-built form and a database will serve you better for the next decade. We would rather tell you which one it is than sell you the exciting version.
If you want to talk through where AI genuinely fits in your operation — and where it does not — take a look at our approach to AI implementation or get in touch.