Measuring AI Quality: A Test Plan for Accuracy, Cost and Latency

It is easy to evaluate an AI feature with a few successful examples. The hard part is seeing for which users a new model or prompt version improves and where it regresses. That takes a small but representative evaluation set that the product team can understand and the engineering team can re-run.

Tie success to an observable outcome

Instead of “the answer is good”, write checkable conditions such as “The request was assigned to the right category”, “The claim in the answer is supported by a permitted source” or “The generated file opens and contains the required fields”. An agent saying it performed an action and the action actually happening in the system are two different things.

Anthropic’s January 2026 evaluation guide covers code-based checks, model-based grading and human evaluation together. For agents that take multiple steps, both the execution transcript and the final state need to be examined. Model-based graders should also be calibrated against human evaluations. Anthropic: Evals for AI agents.

How to prepare the first test file

In the file we recommend, each row contains a user goal, the input, the expected outcome, unacceptable behavior and a severity level. Don’t pick examples only from the questions you tried often during development. Include short and incomplete requests, typos, conflicting information, missing sources and tool outages as well.

• Normal case: The user asks a clear question; the required data is available.
• Edge case: There are two records with similar names; the system must tell them apart.
• Negative case: An unauthorized action or non-existent information is requested.
• Resilience: A tool slows down or the same request arrives again.

Track three metrics on the same task

Quality: How many tasks passed the acceptance conditions? Break failures down by type; a critical data leak should not dissolve into the same score as a minor style error.

Cost: Don’t count only the first model call. Record retries, search, tool use and any human correction time separately. For “cost per successful task”, divide the total cost — with a clearly defined scope — by the number of successful tasks. If there are no successful tasks, don’t report this ratio as a meaningful result.

Latency: Measure the gap between the first response appearing and the task being completed. Alongside the median, track the 95th percentile, which shows the slow requests. What matters to the user is sometimes when the text starts streaming, and sometimes when the downloadable file is ready.

Keep the comparison fair

Run the old and new versions with the same dataset, tool versions and starting conditions. Record the test conditions. Instead of cherry-picking a single successful run, keep the variability between repetitions visible. Separate the examples you used while developing the prompt from the final evaluation examples; otherwise you may simply have fitted to familiar questions.

Leave a short record for the release decision

The decision record should answer four questions: Which version changed? Which tasks improved? Which error types are still open? Under what conditions will you roll back to the old version? Instead of imposing a universal success percentage, set thresholds with the product owner based on business impact.

By making this plan part of your AI-powered test automation process, you can pass model, prompt and tool changes through the same quality gate.