AI agent evaluation: tests beyond a good demo

What should an AI agent evaluation measure?

An agent evaluation should check whether the task succeeded, whether the system respected its allowed actions, and what the run consumed. A convincing final message is insufficient when a tool can change state. Anthropic's agent evaluation guidance distinguishes a task, repeated attempts, grading, the execution trace, and the final environment state. For a portfolio project, make those pieces inspectable before you compare models. The support assistant and scorecard below are proposed practice materials; the numbers are examples, not ProofCohort performance results.

• Source: Anthropic's guide to agent evaluations

Define a small, testable support assistant

Create a fictional workshop booking service. The assistant can read approved policy pages, retrieve a synthetic booking after server-side authorization, and draft a support note. It cannot send messages, change bookings, or issue refunds. Write the policy first: for example, a fictional workshop permits cancellation until a stated cutoff, and exceptions require a human decision. Give the assistant a fixed current time during tests so a date-dependent answer can be reproduced. A narrow tool set makes both correct behavior and boundary violations easier to inspect.

Create a case file before tuning the prompt

Start with around twenty carefully reviewed cases if that is practical for your project. Keep a separate development set for changes and a held-out set for comparison; twenty cases cannot establish production reliability. For each case, record the user request, test identity, starting records, permitted tools, expected evidence, required outcome, and forbidden outcome. A second person should be able to decide whether the result passed without reading your intentions into the answer.

Grade outcomes separately from writing quality

For this assistant, use executable checks for tool names, access decisions, record changes, and required source identifiers. Review whether the cited passage actually supports the response; a citation being present does not prove it is relevant. Use a short human rubric for clarity: does the reply answer the question, distinguish known facts from gaps, and explain the next step? If you use another model to score prose, compare its judgments with human-reviewed examples. Do not allow a fluent explanation to compensate for an unauthorized disclosure.

Record a scorecard with visible denominators

Use one row per attempt with case ID, run ID, agent version, model identifier, document version, outcome pass/fail, boundary pass/fail, duration, tool calls, and measured token usage or cost. Separate provider-reported values from estimates. For a proposed three-attempt run of twenty cases, report both the number of passing attempts out of sixty and the number of cases that passed all three attempts. Neither fraction is a production success rate: the cases are small, selected, and may share failure causes.

• Report failures by category. Missing evidence and broken authorization need different fixes, even when both lower the overall score.
• Record duration for failed and successful attempts. A slow timeout should not disappear from the latency report.
• Mark unmeasured cost as unknown. If estimating from token use and a price sheet, save the pricing date and explain excluded charges.
• Keep the raw attempt log. A summary should let a reviewer trace a failed score back to the input, tool result, and final response.

Compare one change at a time

Freeze the test cases, starting records, corpus, and grading rules before comparing two versions. Change one thing, such as the retrieval method or prompt instruction. Rerun every case rather than selecting the questions that improved. Record regressions alongside improvements. Once you repeatedly inspect a held-out set to tune the system, it has become development material; prepare fresh unseen cases for the next honest comparison. Small score changes are reasons to investigate, not automatic proof of a better agent.

Set a release decision you can explain

For this practice assistant, block the demo if a test exposes another user's booking or falsely claims that an action occurred. Treat these as explicit project requirements, not a universal certification rule. Publish the evaluation runner, synthetic fixtures, grading instructions, result tables, and unresolved failures. Redact secrets and any real conversation data. Your strongest case study is a reproducible explanation of what failed, why you changed the system, and what the next evaluation showed.

• Plan the experiment with a project brief
• Document your evidence and limitations