TL;DR
Builders of tool-calling agents want proof that the agent picks the right tool and passes the right values before it is allowed near a live system. The model only proposes a call; your own code executes it. Put that executor behind a seam, swap in recording fakes for tests, and assert the tool name, the exact arguments and the calls that must never happen. Run each scenario several times and check the fakes against a sandbox, so a passing suite still describes the real API.
In this article
8 min read
For the client-defined tools in this article, neither OpenAI’s nor Anthropic’s API runs the tool for you. The model returns a tool name and a set of arguments, and your application executes them: OpenAI’s function calling guide puts it as “you must execute it and return the result”, and Anthropic’s tool use overview says client tools “run in your application” (both checked 13 October 2026). That handover is where a test takes control. Replace the code that executes tools with fakes that record every call and return fixed results, give the agent a scenario, and check the record with ordinary assertions. The real model still chooses the calls; the test does not touch the real client-defined tools.
Put the switch inside the tool, never in the prompt
The common mistake is to offer the model two tools, a real one and a mock, and tell it which to use during tests. A builder asking on the n8n community forum in August 2025 saw the problem clearly: the mock carries a different name from the tool it replaces, and relying on the model to pick the mock is non-deterministic, so a test run could still call the real tool. They wanted JSON scenarios that list each tool’s mock response and the tools expected to run. A LangChain user asked the same about trajectory tests in July 2025 and was told the evaluation package runs real tools unless you replace them yourself. A September 2026 question on Reddit’s r/LangChain framed it as a business check: does the balance the agent states match the value in the backend?
The fix is one tool with one name whose implementation decides what answers. The model sees the same schema in both environments; only the code behind it changes.
In n8n this is possible without leaving the editor (documentation read 13 October 2026). The Call n8n Workflow Tool loads a sub-workflow as a tool. Inside that sub-workflow, the Evaluation node’s Check If Evaluating operation branches on whether the current run is an evaluation, so the branch can return a fixture during evaluations and call the real system otherwise. Turning on the agent’s Return intermediate steps option adds an intermediateSteps field that later nodes can read to see which tools ran (n8n, fix common issues). Pinned data is no substitute: n8n describes it as a development feature, unavailable in production executions, and only for nodes with a single main output.
One boundary matters before you write any scenario. Provider-run tools, such as Anthropic’s web search or code execution, run on the provider’s infrastructure and never pass through your executor. They cannot be faked at your seam, so keep them out of scenarios that must stay sealed.
Fake the tools to test the model; fake the model to test your code
Two different fakes are easy to confuse. A scripted model replaces the LLM: OpenAI’s Agents SDK ships ScriptedModel in agents.testing (testing guide), and Pydantic AI offers TestModel, FunctionModel and a global switch that blocks real model requests (testing guide). These make your agent loop, retries and tool plumbing deterministic and cheap to test. They say nothing about whether the model chooses well, and OpenAI’s guide says as much: use real provider adapters for behaviour owned by the external model.
So a tool-choice test keeps the real model and fakes the tools. That is the test this article is about.
Write the expected calls down before the run
Take an illustrative agent with three tools, find_customer, get_balance and issue_refund. The scenario is a customer asking for the balance on account C-1042. Before running anything, write down what a correct run looks like: get_balance is called once, its customer_id is exactly C-1042, issue_refund is never called, and the answer contains the balance the fixture returned.
class RecordingTools:
def __init__(self, fixtures):
self.fixtures = fixtures # tool name -> canned result
self.calls = [] # (name, arguments) in call order
def execute(self, name, args):
self.calls.append((name, args))
if name not in self.fixtures:
return {"error": "tool not available in this test"}
return self.fixtures[name]
def test_balance_question_reads_and_never_refunds():
tools = RecordingTools({"get_balance": {"customer_id": "C-1042", "balance": "182.40"}})
answer = run_agent("What is the balance on account C-1042?", tools=tools)
names = [name for name, _ in tools.calls]
assert names == ["get_balance"]
assert tools.calls[0][1]["customer_id"] == "C-1042"
assert "issue_refund" not in names
assert "182.40" in answerDecide argument by argument how strict to be. Identifiers, amounts, currencies, target accounts and permission scopes must match exactly. Free text, such as a search phrase, rarely should. LangChain’s open-source agentevals package (0.0.9) encodes the same choices: trajectory matching can be strict, unordered, subset or superset, and tool_args_match_overrides can name, per tool, the fields that must match exactly (agentevals repository). That is a deterministic check, which is a different kind of evidence from a model grader’s opinion; deterministic proof and model judgment are different evidence explains why the two should not be mixed into one score.
Strict schemas help, but only with shape. OpenAI’s strict: true makes calls adhere to the function schema, and Anthropic offers strict tool use for the same purpose. A well-formed customer_id can still belong to the wrong customer. Anthropic’s overview also notes that a model may infer a plausible value when a required parameter is missing, so add a scenario without the account number and assert that the agent asks for it instead of calling get_balance.
Read repeated runs as a count, and let forbidden calls fail outright
The same scenario can produce a different tool choice on the next run. n8n’s documentation acknowledges this noise and suggests duplicating dataset rows so each input runs more than once. OpenAI’s agent evaluation guide treats the trace, the record of model calls and tool calls for one run, as the thing to grade, with “Did the agent pick the right tool?” among its example questions, and recommends moving to datasets and eval runs when you need repeatability.
Record each scenario as a count: passed 19 of 20, with the failing runs kept for reading. Two kinds of failure deserve different treatment. A forbidden call, such as a refund in a read-only scenario, fails the scenario outright, however rare it was. A harmless variation, such as an extra lookup, belongs in the pass rate. Rerun the suite whenever the model, the prompt or a tool description changes, because each of those can change the choice.
Keep the fakes honest against a sandbox
A fake is written from your understanding of an API. When the API changes, the fake does not, and the suite keeps passing while production calls fail. Martin Fowler’s description of contract tests (12 January 2011) is still the clearest answer: keep running your tests against the double, and separately check that the double returns what the real service would. Run those checks on the rhythm of the external service rather than your own commits; Fowler suggests daily is often enough. A failure means updating the fakes, and probably your code.
Record fixtures from a sandbox, never from production. Where the provider has one, use it: Stripe’s sandboxes are isolated environments in which payments are not processed by card networks, and Stripe recommends separate sandboxes for local development and continuous integration. Where an API has no sandbox, contract-test its read endpoints with a dedicated low-privilege account and keep every write tool faked. A fake that shares the builder’s assumptions cannot catch a wrong assumption, which is the same trap described in an evaluator cannot rescue tests that share its assumption.
If your agent can change records or move money and nobody has written down which calls it must make and must never make, that is a good place to start a conversation. We build and run our own agents in production this way: every tool has a written contract, tool-call scenarios test the real model against recording fakes and mocks with fixtures recorded from a sandbox, and the pass bar is set per tool and signed by the person accountable for the business process that tool touches. No agent we have built for a customer is in production yet. See how we build and run agents.
Frequently Asked Questions
How do I test that my AI agent calls the right tool with the right arguments without touching real systems?
Swap the code that executes tools for recording fakes, run the real model against written scenarios, and assert the call log. First list every tool that executes outside your seam: provider-run tools, remote MCP connectors and workflow nodes that call an API directly. Those need a sandbox, because a fake at your seam never sees their calls.
When a test fails, how do I tell whether the agent chose the wrong tool, used the wrong arguments or got a bad result?
Read the call log in that order. A wrong name is a choice failure; a right name with a mismatched field is an argument failure. Because fixtures are fixed, a bad tool result in a test can only come from the fixture, so put a distinctive value in each one and check whether the answer used it. Bad results from the real tool belong to the contract tests.
Which tool calls must match exactly, and where should the test allow different orders or phrasings?
Exact: identifiers, amounts, currency, target account and permission scope, compared after normalising formats (182.4 and “182.40” are the same amount). Flexible: search phrasing and wording. Enforce order only where a later call depends on an earlier result, such as finding the customer before reading their balance; independent lookups can run in any order.
If I swap real tools for mocks, how do I stop the mocks drifting from the real APIs?
Store each fixture with the API version and the date it was recorded, regenerate it from the sandbox on a schedule, and fail the contract check when a field appears, disappears or changes type. Where the provider lets you pin an API version in requests, pin it in both the agent and the fixture recorder.
How many runs per scenario, what pass rate is enough, and who signs off?
No published threshold fits every tool. Zero failures in n runs still leaves an upper 95% bound on the failure rate of roughly 3 divided by n (Hanley and Lippman-Hand, JAMA, 1983), so twenty clean runs only suggest a rate below about 15%. Set the bar per tool by what a wrong call costs, and have the person accountable for the business process that tool touches sign it off, recorded with the model and prompt versions.
