An agent that works in a demo can still fail on the tenth real customer. The difference is evaluation: a repeatable way to measure whether the agent does the job, on your cases, before and after every change. This is the process we follow on every build.
1. Build the test set from real history
Pull a few hundred real cases: tickets, questions, documents, whatever the agent will face. Include the easy ones, the hard ones and the ugly ones. For each, write down what a correct outcome looks like. Real history beats invented examples because it contains the messiness your agent will meet.
2. Define what "pass" means before you test
Agree thresholds in advance with the business owner, for example: correct resolution on at least a stated share of cases, zero actions outside permissions, and every escalation carrying a usable summary. Deciding after seeing results invites wishful thinking.
3. Test each layer separately
- Retrieval: does it find the right documents or records?
- Tool use: does it call the right tool with the right parameters, and refuse when it should?
- Final answer: is it correct, grounded in sources and in the right tone?
- Escalation: does it hand off when it is unsure, with context?
4. Attack the edge cases
Test ambiguous requests, missing data, contradictory records, attempts to push the agent outside its permissions, and prompt-injection text inside documents or tickets. These are the cases that cause incidents.
5. Keep a human in the loop for high-impact actions
Refunds above a limit, changes to customer data and external messages should require approval until the evidence shows the agent is reliable. Loosen permissions as the numbers justify it.
6. Run the suite on every change
Prompt edits, model upgrades and new tools can all cause regressions. An automated evaluation run makes those visible immediately. It also makes model swaps safe, because you can compare providers on your own cases. See which model should power your agents.
7. Monitor after launch
Log every decision, sample live conversations, track resolution and escalation rates, and feed new failure cases back into the test set. For ongoing monitoring and improvement, that is what our AgentOps retainer covers.
Evaluation is built into every fixed-price agent build we deliver, and the suite is handed over with the code so you can keep running it.
Frequently asked questions
How many test cases do you need to evaluate an AI agent?
A few hundred real cases is a practical starting point for a single workflow, covering common, hard and edge cases. More important than the number is that they come from real history and are labelled with correct outcomes.
What is a good pass threshold for an AI agent?
It depends on the cost of mistakes. Agree it with the business owner before testing: for example a minimum correct-resolution rate, zero out-of-permission actions, and usable escalation summaries.
Do you need to re-evaluate after every model update?
Yes. Prompt changes, model upgrades and new tools can all change behaviour. An automated evaluation run on every change is the safest way to catch regressions.
Related: AgentOps · AI agents explained
Want a straight answer for your project?
Book a 30-minute call. I will tell you what I would do, and a fixed price if it makes sense.