Evals
Define quality before the demo persuades you the system is ready. · AI Product Management · Lesson 50 · 4 min
Evals · 4 min
Situation
“It looked good in my testing.”
A team tries an AI support assistant on ten familiar questions. The answers sound helpful. In production, users ask incomplete questions, refer to old policies, and include unusual account conditions.
A handful of favorable examples shows possibility. It does not establish a reliable operating range.
Mental model
An eval is a repeatable test of intended behavior.
Use a test set of representative inputs and a rubric describing acceptable outputs. Include routine tasks, important edge cases, missing evidence, and cases where the system should refuse, clarify, or escalate.
Keep a held-out set for checking changes. If you tune every prompt against the same examples, you can improve the score while learning little about general performance.
Example
Make the rubric observable.
For a policy assistant, assess whether the answer uses the applicable policy, supports claims with evidence, avoids invented commitments, and gives the user a useful next step.
Separate fatal errors from style issues. A friendly tone should not compensate for revealing another customer's information or promising an unavailable refund.
Worked case
Evaluate the difficult answer.
Input: “I bought it last month but lost the receipt. Can you refund it?” The supplied facts contain no purchase date or applicable policy.
A passing response identifies the missing information and explains the next check. A failing response confidently says yes or no without evidence. The rubric should make that distinction explicit enough for reviewers to apply consistently.
Methods
Combine human judgment with scalable checks.
Code can check required fields, allowed tool calls, and exact constraints. Humans can assess usefulness and subtle errors. Model-based graders can scale some judgments, but need calibration against trusted human review.
Review disagreement between evaluators. A grader is another imperfect system; do not treat its score as ground truth merely because it is automated.
Coverage
Measure the distribution of failures.
Report results by task type and severity, not only one aggregate score. A 95% pass rate can hide complete failure on a small but important language or account category.
Estimate uncertainty when samples are small. Add real failure cases to future regression tests while keeping a fresh evaluation set to avoid overfitting the benchmark.
PM decision
Use evals as a release gate and diagnostic tool.
Compare the complete system before and after changes to the model, prompt, retrieval, or tools. Track quality, latency, and cost per successful task.
Define which failures block release and who owns review. After launch, monitor drift and sample real outcomes with appropriate data handling. A prelaunch suite cannot cover every future input.
Remember this
Quality is behavior you can describe and inspect.
Evals turn “seems better” into evidence about a task, dataset, and rubric. They support judgment; they do not remove the need to decide which errors are acceptable.