Look at the data: evals for AI products

Ask a team how they know their AI feature works and the honest answer is often: it looked good when we tried it. That is a vibe check, and vibe checks do not survive contact with real users, who phrase things differently, bring messier data and find the edge cases on the first day.

The alternative is an evaluation practice, usually shortened to evals. Some of the most practical writing on the subject comes from people who have built these systems for many teams. Hamel Husain’s guide is a good example, and its central claim surprises people: the most important activity in evals is not building metrics. It is error analysis, which means looking at the data.

Why generic metrics mislead

It is tempting to start with an off-the-shelf score for helpfulness, relevance or similarity to a reference answer. These numbers are easy to compute and easy to put on a dashboard. Husain is blunt about them: “Generic evaluations waste time and create false confidence when you use them as quality measures.” They measure abstract qualities, while your users experience specific failures: a wrong date, an ignored instruction, a confident answer to a question that needed a follow-up.

Step one: read a hundred conversations

Error analysis is simple to describe and easy to skip. Collect a representative set of real interactions, a hundred or more. Have someone who understands the users read at least thirty of them closely and write a plain note about what went wrong in each. Then group those notes into a small set of failure types and count them.

The counting is what makes it useful. After an afternoon you no longer have a vague sense that quality is “mostly fine.” You have a ranked list: this failure happens in a fifth of conversations, that one in two percent. The ranking tells you what to fix first, and what is worth measuring at all.

One person decides what good means

Quality judgements need an owner. Husain recommends a single domain expert, someone who deeply understands the users, as the final word on what passes. A committee of annotators produces disagreement and slow decisions. One accountable person produces a consistent standard that can be written down and, later, taught to an automated grader.

Pass or fail, never one to five

Rating outputs on a five-point scale feels more precise than a yes or no. It is not. As the guide puts it, “the difference between adjacent points (like 3 vs 4) is subjective and inconsistent across annotators.” A binary judgement forces a clear decision about whether the output did its job. If you want more resolution, write more binary checks for specific things rather than finer grades for vague ones.

Expect your criteria to move

Researchers studying how people grade model outputs found something every product team will recognise. In “Who Validates the Validators?”, Shreya Shankar and colleagues describe criteria drift: people need criteria to grade outputs, but grading outputs is how they discover their criteria. The standard sharpens as you see more data. Treat the eval set as a living document, not a contract signed on day one.

Automate only what you will fix repeatedly

Not every failure needs an automated test. Many are fixed once, by a better prompt or a missing feature. For the ones that recur, start with plain code checks: is the date in the right format, did the answer cite a source. Reach for a model acting as judge only when the check genuinely needs judgement, and validate that judge against the expert’s labels before trusting it. In Husain’s words, “what ultimately matters is how well your judge aligns with human judgments.”

Where design comes in

The most interesting result of error analysis is how many failures are not model failures at all. People ask ambiguous questions because the interface invited them to. They miss the caveat because it was in the wrong place. They get a poor answer because the product never asked for the context it needed. Reading real conversations is the fastest user research an AI team can do, and its fixes are often in the interface, not the prompt.

You cannot improve what you have not read. Start with the data, not the dashboard.

A minimal loop to start this week

• Log every interaction with its inputs, context and output.
• Every week, have the domain expert read thirty and note what went wrong.
• Group the notes into failure types and count them.
• Fix the top one; add a pass or fail check only if it can come back.
• Run the checks on every prompt or model change, like any other test.