There is a pattern we now expect at the start of almost every engagement. A team has built something on top of a language model, it works in the demo, and it works in the hands of the three people who built it. Then it reaches real users and the reports start arriving: it is wrong sometimes, nobody can say how often, and nobody can say whether last week’s prompt change made it better or worse.
The diagnosis is almost always the same. The team has a product and no measurement. What follows is not a tooling problem. It is that the evaluation was treated as overhead on the way to shipping rather than as the thing being shipped.
You do not own the model. You own the definition of correct.
The only asset that survives
Consider what is actually durable in a system built on a hosted model. Not the model — it will be deprecated, repriced, or quietly updated underneath you. Not the prompts — they are rewritten every time the model changes, and a prompt tuned against one version is an artefact of that version. Not the orchestration code, which is thin by design and tends to get rewritten as the surrounding product moves.
What survives is the set of cases you have decided the system must handle, paired with a defensible account of what handling them correctly looks like. That artefact outlives every component around it. When the next model arrives, a team with a good evaluation suite runs it and has an answer in an afternoon. A team without one runs it, reads a handful of outputs, and forms an impression.
What makes a suite honest
Assembling cases is the easy half. The harder half is keeping the suite pointed at the thing you care about rather than the thing that is convenient to score. A few habits do most of the work:
- Measure the grader. If a model is judging outputs, its agreement with human judgement is itself a number you should know. An unvalidated grader is a confident random number generator.
- Keep the failures. Every real defect that reaches a user becomes a case. This is the cheapest source of signal available and most teams throw it away in a ticketing system.
- Version the data. A score is meaningless unless you can say which set produced it. Suites drift, and undocumented drift looks exactly like progress.
- Write down what it does not cover. Every suite has blind spots. Naming them is what separates a measurement from a reassurance.
The signature failure
The failure worth watching for is a suite that improves steadily while the product does not. When we see this, the cause is nearly always the same: over months, cases that were hard to score got quietly dropped, and cases that were easy to score got added. Nobody decided this. It is the natural gradient of a team optimising a number.
The correction is to periodically re-derive the suite from user reality rather than from the previous suite — sample recent production traffic, look at it directly, and ask whether the distribution in the evaluation set still resembles the distribution in the world. It usually does not, and the gap is the finding.
A reasonable objection
The objection we hear is that building this first slows everything down, and for a prototype that is true. Before there are users there is nothing to be right about, and a rigorous suite is premature. The transition point is narrow and easy to miss: it arrives the moment someone outside the building depends on an output. After that, every week without measurement is a week of decisions made on impressions, and impressions compound into a system nobody is willing to change.
The teams that move fastest in this space are not the ones that skipped the measurement. They are the ones that built it early enough that changing the system stopped being frightening.