How we evaluate agents before they meet customers

Answer: We evaluate agents before they meet a customer by running 50 to 200 graded test cases pulled from the workflow’s real history, including the ugly ones, and scoring the agent against a pass threshold written into the specification before the build starts. If it doesn’t clear that bar, it doesn’t launch.

Why does a demo passing mean almost nothing?

A demo proves an agent can succeed once, on a case someone picked to show it off. Production asks a different question entirely: how often does it succeed, and on what? Those are not the same claim, and the gap between them is where most agent projects quietly fail after the kickoff meeting goes well. A stakeholder watches a clean run, signs off, and three weeks later the agent is misrouting 1 in 6 tickets because nobody tested it against the messy 15% of cases that make up most of a support queue’s actual variance.

The fix is not a better demo. It’s refusing to call a demo evidence in the first place.

What actually goes into an evaluation set?

We build the evaluation set from your workflow’s real history, not synthetic examples written to make the agent look good. That means 50 to 200 graded cases at launch, drawn directly from what actually happened: the routine cases, the edge cases, and the ones your team already knows are hard. Each case gets a graded expected outcome, not a vague “did it seem reasonable” judgment call.

This matters because synthetic test cases tend to cluster around what the person writing them already expects the agent to handle well. Real historical cases don’t have that bias. They include the ticket where the customer’s request contradicted their account data, the invoice with a currency mismatch, the support message written in 2 languages in the same paragraph. If the evaluation set doesn’t include those, the agent’s score is measuring the wrong thing.

How often should an agent be re-evaluated after launch?

3 moments, not 1. Before launch, evaluation runs as a hard gate: the system does not go live if it doesn’t clear the threshold. On every change to the agent’s prompt, tools, or underlying model, the same evaluation set runs again as a regression check, because a change that improves one case can silently break another. And on a schedule, independent of any change your team makes, because the model provider underneath the agent can update the model without asking, and that update can shift behavior on cases that used to pass cleanly.

That third moment is the one teams skip most often, and it’s the one that catches the failures nobody was watching for.

What happens to a case the agent gets wrong in production?

It becomes a permanent regression case. Every caught failure gets added to the evaluation set, which means the set only grows over time. This is the mechanism that makes an agent’s reliability trend upward month over month instead of oscillating between “seems fine” and “why did it just do that” depending on which version happens to be live. An evaluation set that never grows is a sign nobody is watching production closely enough to catch what’s actually breaking.

Where is the pass threshold decided, and by whom?

In the specification, before a single line of the build starts, not argued about after 8 weeks of work are already sunk into it. Writing the threshold down early forces an uncomfortable but useful conversation: what error rate is actually acceptable for this specific workflow, given what happens when the agent is wrong. A support-ticket triage agent and a billing-reconciliation agent do not get the same threshold, because a wrong answer costs something different in each case. Deciding that in week 8, after the team is emotionally and financially invested in shipping, is how thresholds quietly get lowered to match whatever the system happens to score.

This is the same evaluation discipline we run on every agentic engineering build: an evaluation set drawn from your real history, 3 moments where it runs, a pass threshold fixed before the build starts, and a regression set that only grows. If your team is trying to decide whether an agent is ready to meet a real customer, this is the test we’d run before saying yes.