How to evaluate an AI application before production

An AI demo usually shows a few inputs chosen by its builder. A production decision needs a broader question: does the application behave usefully on representative tasks, including the ones it should decline or hand off? You can begin with a small evaluation set. The key is to define what success means before looking at the model’s answer, then keep the experiment repeatable enough to compare changes.

Define a task and its expected outcome

For a fictional support assistant, a task might be “Explain whether this order qualifies for the published return process.” The expected outcome could require the correct policy reference, the relevant order status and a clarification when the purchase date is missing.

Write checks for the actual result. An assistant saying “I updated the ticket” is not proof that the ticket changed. If the workflow includes tools, inspect the resulting state and the permitted scope of the action. This is why evaluations may need both output checks and environment checks.

Reference: Anthropic — Demystifying evals for AI agents ↗

Build a balanced starting set

Include ordinary requests, missing information, conflicting records, unavailable tools and out-of-scope requests. For retrieval, record the source that should be found. For a business action, record whether the application should proceed, ask or stop.

Keep a separate held-out set that you do not repeatedly use to tune the prompt. The first set helps development; the held-out set helps reveal whether you only improved familiar examples. The exact size depends on the task and risk, so treat a small initial set as a baseline rather than statistical proof of reliability.

Use deterministic checks where they fit

A schema check can establish that a response is parseable and contains the expected fields. A source-ID check can establish that a named document was retrieved. Neither proves the explanation is factually supported, so give each check a narrow meaning.

The Python example below grades two fictional responses against a single expected source ID. Save it as grade.py and run it with Python 3. It reports 1 of 2 matches. This is a test of the grading logic only; it does not query a model or establish the quality of an AI system.

cases = [
    {"id": "return-policy", "expected_source": "policy-returns",
     "response": {"source_ids": ["policy-returns"]}},
    {"id": "missing-evidence", "expected_source": "policy-delivery",
     "response": {"source_ids": []}},
]

def has_expected_source(case):
    ids = case["response"].get("source_ids", [])
    return case["expected_source"] in ids

passed = sum(has_expected_source(case) for case in cases)
assert passed == 1
print(f"Source match: {passed}/{len(cases)}")

Review meaning and failure severity

Ask a knowledgeable reviewer whether the answer follows the available evidence, covers the request and handles uncertainty appropriately. Record disagreement so you can improve an ambiguous rubric rather than forcing an arbitrary score.

Group failures by consequence. A slightly awkward sentence is different from exposing another customer’s record. A model-based grader can help scale review, but calibrate it against human judgments and inspect examples where they disagree. Avoid hiding severe failures inside an attractive average.

Reference: Anthropic — Demystifying evals for AI agents ↗

Measure the cost of completing a task

Record completion time, retries and resource usage alongside quality. A response that passes only after several attempts has a different operational cost from one that works on the first try. Repeat selected cases to see how variable the behavior is.

Before releasing a change, compare it with the current baseline on the same cases. Keep clear gates for critical failures, and monitor real usage after launch. Add newly discovered failures to the suite without silently changing the meaning of older results.