TL;DR
An agent that has impressed in internal demos is about to meet customers or staff, and a good average score does not tell you whether it is safe to let it act. A useful test plan ranks the failures that would hurt someone, tests the code and the model’s judgment separately, repeats every scenario, and ends with a small watched pilot. Release when no blocking failure appears in any layer and a named person has accepted the remaining risk in writing.
In this article
8 min read
A test plan for an AI agent should cover four things before real users see it: the failures that would actually hurt someone, ranked by severity; ordinary code tests for the tools, permissions and limits; scenario runs that score the model’s decisions, repeated several times each and including awkward and hostile users; and a small pilot with stop conditions and a way back. It is ready when no layer shows a blocking failure, lesser failures sit behind human approval, and a named person has signed for what remains.
The question keeps being asked in public. In July 2023 a developer on the OpenAI developer forum described an agent with function calling and a first automated check: ask every question in a bank three times, count full and partial successes, score how similar the answers are, then ask how others make sure end users get consistent behaviour. In August 2026 a Product Hunt discussion asked which signals give confidence that an agent is reliable enough for production, and a September 2026 thread on Reddit’s r/LLMDevs asked simply how people test agents before production. All three ask about tests. None of them gets as far as the decision the tests are for.
Rank the failures by harm before writing a test
Start with a list of what could go wrong, sorted by who gets hurt and whether it can be undone. The ranking decides everything after it: which scenarios you write, how strict each pass mark is, and who has to sign.
Take an illustrative staff-facing IT helpdesk agent, not a real deployment, that answers questions, resets passwords, grants access to shared folders and hands security incidents to a person. Its plan might start like this:
| Failure class | Examples for this agent | Release rule | Who accepts the risk |
|---|---|---|---|
| Blocking | Access granted beyond the requester’s role; a password reset without the identity check; another employee’s details shown; a reported phishing email not handed to security; obeying instructions hidden in a forwarded message | Not one failure in any repeated run | IT operations lead, with the security officer for access and personal data |
| Approval | Wrong troubleshooting steps; a policy quoted inaccurately; an access grant within policy that a person would have refused | Live only behind a person who approves the action or the answer first | Service desk lead |
| Cosmetic | Awkward tone, odd formatting, an unnecessary extra question | Logged and fixed in a later release | Product owner |
The owners should be the people who would clean up the damage. If nobody will put their name in the last column, the agent is not ready for that action: remove it or put a person in front of it.
Three layers, each with a different blind spot
Each kind of testing misses something the others catch.
Code tests cover everything that is ordinary software: the tools the agent can call, their input checks, permissions and hard limits. Which folders a role may receive belongs here, enforced in code, because a rule written only into the prompt is a request the model may ignore. OWASP’s 2025 entry on excessive agency names the three root causes these tests should close: more functionality, more permission or more autonomy than the task needs. How to test tool calls without touching real systems is a technique in its own right and gets its own article in this series.
Scenario runs score the model’s judgment: did it pick the right action, refuse the wrong one, escalate when it should? OpenAI’s evaluation best practices (checked 9 October 2026) recommend datasets that mix typical, edge and adversarial cases, and warn against “vibe-based” evaluation. Anthropic’s engineering guide Demystifying evals for AI agents (January 2026) suggests starting with 20 to 50 tasks drawn from real failures and reading the transcripts before trusting any score. Graders can be code, a model or a person; a model grader needs checking against human judgment first. A passing unit test and a model’s verdict are different kinds of evidence, and the release record should keep them apart.
A watched pilot puts the agent in front of real people and real data, which no scenario set fully imitates.
Repeat every scenario and set the bar per class
The forum author’s instinct to ask each question three times was sound, because the same agent can pass a scenario on Monday and fail it on Tuesday. The τ-bench paper by Yao and colleagues (arXiv, June 2024) introduced pass^k, the chance that an agent succeeds on all k attempts at the same task, and found that function-calling agents of that time solved fewer than half the tasks and passed fewer than a quarter of retail tasks eight times in a row. Anthropic’s guide gives the arithmetic: an agent that succeeds 75% of the time on a task has about a 42% chance of succeeding three times in a row.
That changes how the bar should be set:
Decision guide
- If a scenario tests a blocking failure, every repeat must pass. One failure in ten runs shows that the scenario is not reliably safe; the production failure rate is still unknown.
- If it tests an approval-class failure, a threshold is acceptable as long as a person reviews that action in production until the rate is known.
- If it tests a cosmetic failure, track the rate and let it improve over releases.
An average across the whole suite hides exactly the failures that matter. A high overall score can still contain an access-grant scenario that fails one time in five.
Run the pilot as the last test
Google’s SRE workbook defines canarying as “a partial and time-limited deployment of a change in a service and its evaluation” (Canarying releases). A first agent release deserves the same treatment: a limited group of users or cases, a fixed period, and an evaluation plan written before it starts.
Write down three things before the pilot opens. First, which actions run unsupervised and which wait for a person: in the example, every access grant might need approval during the pilot, whatever the scenario results said. Second, the stop conditions, such as any blocking-class failure, or approval-class failures above an agreed rate. Third, the way back, tested in advance: switching the agent off, routing conversations to people, and undoing anything it did.
Every failure the pilot finds goes back into the scenario set, so the next release is tested against it. Measuring correctness once the agent is in full production, and noticing when a model update changes its behaviour, are separate problems with their own articles in this series.
Write down who decided, and on what evidence
The release decision should survive the person who made it. A short record is enough:
- The agent’s job, its allowed actions and the acceptance criteria agreed before building.
- The failure ranking with an owner for each class.
- The versions tested: model, prompt, tools and scenario set.
- Results per layer, with repeat counts for each scenario.
- The approval-class failures still open and the human approval that covers each.
- The pilot’s stop conditions and the tested way back.
- The approver’s name, the date and what they accepted.
A signature only means something if the person could have said no. An approver who receives a green summary an hour before launch is approving without authority; give them the failure list and the open risks along with the score.
Some of this record is also law. Since 2 August 2026, Article 50(1) of the EU AI Act has required providers to design AI systems that interact directly with people so those people are told they are dealing with AI, unless that is obvious, which makes the disclosure a test case like any other. If the use case is high risk under Annex III, such as credit or recruitment, Article 9(8) requires testing before release against prior defined metrics and probabilistic thresholds; the 2026 Digital Omnibus amendment moved those obligations to 2 December 2027 (European Commission AI Act Service Desk, consolidated text as at 27 July 2026). Whether your agent is high risk is a question for your compliance owner.
This is how Eliyce builds and runs its own agents: the job, success criteria and acceptance checks are agreed before building, code tests and evaluations of the model’s judgment are kept separate, and every change is specified, reviewed, checked and recorded. We run agents in production today, with sub-agents and retrieval doing real work, so this is the discipline we hold ourselves to, not a theory. We have not yet put an agent we built for a customer into production, so we show the method, not a customer track record. If an agent of yours is close to real users, see how we build and run agents.
Frequently Asked Questions
What should a test plan for an AI agent cover before it reaches real users?
It should cover harm-ranked failures, code tests, repeated scenario runs and a watched pilot, and it should also say what was deliberately left untested. A plan that lists its own gaps lets the approver judge the risk instead of assuming the green parts cover everything.
Which of my agent’s failures would actually hurt a customer, a colleague or the business, and which are only cosmetic?
Ask two questions of each failure: can it be undone, and does it touch someone else’s money, data or rights? Anything irreversible or involving a third party is serious. The people who handle complaints and reverse mistakes today usually know the list already, so start with them rather than the build team.
Which failures must block release regardless of a good average score, and which can go live behind human approval?
A failure blocks when no person would see it before it took effect, such as access already granted, money already paid or data already shown. If a person can see the action and stop it first, it can go live behind approval, provided that person has the time and authority to refuse.
How do code tests, scenario runs and a pilot fit together, and what does each miss?
Code tests miss judgment, scenario runs miss the users and data you did not imagine, and a short pilot misses rare events. Keep them independent where you can: scenarios written by someone other than the person who wrote the prompt find more, because they do not share the builder’s assumptions.
What written evidence should exist at release so someone can later see why the agent was judged ready?
The record in the checklist above, stored with the versions it tested. When the model, prompt or a tool changes, the next release is compared against it, so the record also becomes the baseline for every later change.
