How do I know my agent’s answers are right in production without an answer key?

TL;DR

Once an agent is live, uptime and latency dashboards say nothing about whether its answers are right, and there is no answer key to compare against. Build a partial one from three sources: check every output that a system of record can confirm, have people grade a small random sample against a written rubric, and let a model grader extend that sample only after it agrees with your reviewers. The intended end state is a few quality numbers, with sample sizes, that an owner can track and act on.

In this article

1. Start with what a system of record can confirm
2. Build a small answer key from a random sample
3. Let a model grader extend the sample only where it agrees with your reviewers
4. Treat disagreement, escalation and silence as signals
5. Report a few numbers and route every failure back into the evaluations
6. Frequently Asked Questions

8 min read

To measure correctness, decide what each answer can be checked against. Code can compare some answers with system records; people must judge others. Use a model grader only where it agrees with those reviewers. Other signals are proxies for finding cases, not correctness scores.

Two public questions show where teams get stuck. In March 2026 someone asked on Hacker News how others monitor AI features in production: their tools reported whether the API was up and how fast it answered, but a mediocre answer threw no error, so they could not tell whether users were satisfied. In June 2024 a developer asked on the OpenAI community forum how to evaluate chat responses without a dataset of expected outputs, since the responses vary from run to run and public benchmarks say little about one specific chatbot.

Start with what a system of record can confirm

Many agent outputs make a claim that something else can verify after the fact. If the agent tells a customer their refund was issued, the payments system either holds that refund or it does not; a booking either exists at the right time or it does not. For outputs with a verifiable record, ordinary code can check every eligible conversation and produce a failure rate for that subset rather than an impression.

Three kinds of check cover most of it:

• State checks. Compare what the agent said it did with the final state of the record it acted on: the order, ticket, booking or refund.
• Grounding checks. For answers built from retrieved documents, code can confirm that a cited passage exists; judging whether it supports each claim needs human review or a model-based check. The Ragas paper describes faithfulness measures of this kind that need no human reference answers (Es et al., arXiv, September 2023); they judge support for a claim, not whether the source document itself is correct.
• Later corrections. A reopened ticket, a reversed transaction, an edited draft or a customer who asks the same question again within a few days is a delayed outcome signal you can join back to the conversation.

What remains is the part no record can confirm: whether advice was sound, the tone fitted or something was missed. Separate the two classes explicitly, because a deterministic result and a model’s judgment are different kinds of evidence and should never be added into one score.

Build a small answer key from a random sample

For judgment calls, people are the reference. Draw a random sample of real conversations each week, add a few deliberately chosen ones (escalations, long conversations, new topics), and have a reviewer grade them against a rubric written in advance: correct, partly correct with a stated flaw, or wrong. Count the chosen ones separately: only the random ones estimate a rate.

Sample size decides how much a number can tell you. With 10 failures in 100 randomly sampled conversations, a 95% Wilson interval for the true failure rate runs from about 5.5% to 17.4%; with 40 in 400 it narrows to about 7.4% to 13.3% (method in the NIST/SEMATECH e-Handbook of Statistical Methods). Zero failures in 100 still allows a true rate of up to about 3.7%. So report the count and the interval with every rate, and do not read a two-point change week to week as movement.

Check the reviewers against each other first. Have two people grade the same 30 or so conversations independently and record how often they agree. Low agreement means the rubric is unclear; fix that before any automation. Expect the rubric to change once grading starts: a study of evaluation tooling found that people need criteria to grade outputs but refine those criteria by grading, which the authors call criteria drift (Shankar et al., arXiv, April 2024).

Stored conversations are usually personal data. The GDPR’s purpose limitation, data minimisation and storage limitation principles (Article 5(1)(b), (c) and (e)) apply to a review set as much as to the agent, so agree its purpose, readers and retention with whoever is responsible for data protection before you store anything; this article is not legal advice.

Let a model grader extend the sample only where it agrees with your reviewers

A model grader scores far more conversations than people can read, and the research supports using one with care. In one widely cited study, strong model judges agreed with human preferences more than 80% of the time, about the level at which humans agreed with each other, while showing position, verbosity and self-enhancement biases (Zheng et al., arXiv, June 2023). Later work is less comfortable. Swapping the order of two answers was enough to change which one a grader preferred (Wang et al., arXiv, May 2023); models can recognise and favour their own output (Panickssery et al., arXiv, April 2024); even the best judges in one comparison stayed well behind agreement between humans (Thakur et al., arXiv, June 2024); and across 20 tasks reliability varied so much that the authors conclude graders must be validated against human judgments first (Bavaresco et al., arXiv, June 2024).

The vendors agree in their documentation, checked on 9 October 2026. OpenAI’s evaluation best practices advise scaling a model judge only once it consistently agrees with human annotations. Anthropic’s guide to building evaluations recommends detailed rubrics, a single clear verdict and, generally, a different model to grade than the one that produced the output.

In practice that means:

A grader built on the same model, instructions and retrieved context tends to share the agent’s blind spots, rating a confident wrong answer as correct for the reason the agent gave it. An evaluator cannot rescue checks that share its assumption; independent evidence has to come from somewhere else, which is what the state checks and the human sample provide.

Treat disagreement, escalation and silence as signals

These signals do not measure correctness, but they tell you where to sample next.

• Disagreement. Review first where the grader and your reviewers disagree. Inconsistency also shows inside the agent: SelfCheckGPT found that when a model’s answers to the same prompt diverge across samples, the content is more likely to be invented (Manakul et al., arXiv, March 2023).
• Escalation. The share of conversations handed to a person should be watched in both directions. A sudden drop can mean the agent now answers what it should pass on.
• Feedback. A thumbs-down rate counts only the people who click. Users who leave unhappy and silent are invisible to it, so use feedback to choose what to review, never as the quality number itself.

Each can also move for reasons unrelated to quality, such as a new customer segment, so they trigger a look at the sample rather than a conclusion.

Report a few numbers and route every failure back into the evaluations

Google’s SRE book advises choosing a handful of representative indicators rather than everything you can measure (Service Level Objectives). For an agent, three usually carry the story: the failure rate from eligible state checks with its denominator, the pass rate on the random human sample with its count and interval, and the escalation rate. Name one person who owns them and decides what happens when they move. NIST’s AI Risk Management Framework asks for the same in MEASURE 2.4: monitor the system’s functionality and behaviour in production (NIST AI 100-1, January 2023).

Every confirmed failure, from any source, becomes a new case in the evaluation set that runs before the next prompt, model or tool change, so the answer key grows from real traffic. Observability tools carry much of the mechanics, such as LangSmith’s sampled online evaluators (documentation) and Langfuse’s annotation queues for calibrating a grader against human scores (documentation); the definition of correct still has to come from you.

This only works if “correct” was defined before launch. Eliyce agrees the task, success criteria and acceptance checks before an agent is built, designs actions so they can be verified in code against the real system, and builds each agent so that what goes wrong in production becomes a new evaluation case. Monitoring itself runs on established tooling chosen per agent, Eliyce promises no accuracy rate, and no agent Eliyce has built is in production for a customer yet. If you run an agent whose answers nobody can currently vouch for, talk about your agent.

Frequently Asked Questions