Most teams ship an agent the same way they ship a feature: build it, try it a few times, watch someone nod, push to production. That works for a button. It does not work for a system whose output changes every time you touch the prompt, swap the model, or add a tool. If you cannot say what percentage of a fixed test set the agent gets right today versus last week, you are not shipping software. You are shipping a demo that happened to work when you were watching.
Why manual testing fails here
Manual testing fails because agents are non-deterministic and the failure modes are invisible until volume. Running the same prompt five times can produce five different tool-call sequences, and a change that fixes one case can silently break three others you did not think to check. A human tester notices the case in front of them; they do not notice the ten cases they forgot to run, and they never notice the same case degrading slowly over three model updates. An eval set notices, because it runs the same fixed inputs every time and reports a number, not an impression.
What a minimum eval set actually contains
A minimum eval set is 30 to 100 real examples, each with an input and a known correct or acceptable output, pulled from actual use cases rather than invented ones. Real support tickets, real documents, real user requests — not the clean, cooperative test cases engineers write for themselves. Include the annoying ones: ambiguous requests, missing information, adversarial phrasing, edge cases where the right answer is “ask a clarifying question” rather than act. If your eval set only contains cases the agent was obviously designed to handle, it will tell you nothing you did not already believe.
Each example needs a grading method, not just an answer. For factual or structured tasks, exact match or a rule-based check works and is cheap to run. For open-ended output, you need either a human rubric or a second model acting as judge, and the judge prompt itself needs its own small validation pass, because a bad grader is worse than no grader — it gives you false confidence at scale. This is the part teams skip, and it is the part that makes the whole exercise trustworthy or not.
The gate before launch
The gate is simple: no deployment change ships unless it holds or improves the pass rate on the full eval set, and every regression gets logged with the specific case that broke. This turns “the new prompt seems better” into a number you can defend, and it turns “did we break anything” into a five-minute check instead of a guess. Teams that build this gate early move faster later, because they stop re-discovering the same bugs after every model upgrade. Teams that skip it accumulate a pile of undocumented tribal knowledge about what the agent used to be able to do, which nobody can verify and which erodes every time someone touches the system.
This is the same discipline that shows up in the workflows we build for clients — the eval set is not a side project, it is the thing that makes iteration safe. Without it, every change to an agent is a bet with unknown odds. With it, the bet has a number attached, and the number moves in a direction you chose rather than one you discovered after a customer complained.
What to measure once it is live
Once an agent is in production, the eval set stops being the only signal but it does not stop mattering. Add live monitoring: task completion rate, escalation rate to a human, and a sample of live transcripts pulled weekly and scored against the same rubric used in the eval set. The gap between eval performance and live performance is itself information — a large gap usually means your eval set does not resemble real traffic closely enough, which means it is time to add new cases from production back into the set. This loop, eval to production to eval again, is the actual maintenance cost of running an agent, and it is ongoing, not a one-time setup task.
Who should not build agents yet
If your team cannot commit to writing 30 real test cases before launch, you are not ready to run an agent unsupervised, and you should keep a human in the loop on every output until you are. That is not a failure — plenty of workflows are fine with a human review step permanently attached. But calling a reviewed-by-nobody system “autonomous” because it looked fine in a demo is how teams end up debugging a production incident with no idea whether the agent has ever behaved this way before. We walk through this setup with clients in our services engagements, and the first deliverable is almost always the eval set itself, before a single line of the actual agent logic changes. The measurement comes first because everything else is a guess without it.