Loop Engineering 3: an evaluator cannot rescue tests that share its assumption

Last updated October 14, 2026

7 min read

An evaluator cannot rescue tests built from the same mistaken assumption as the code; it needs requirement-derived examples, external invariants, adversarial cases, or accountable human decisions that exist outside the generated solution. This question matters because faster generation moves work into decisions that still require evidence and ownership.

A fictional loop writes a date rule with the wrong timezone, generates tests in the same timezone, and asks a reviewer using the same context to judge it. Three correction turns improve naming and coverage while preserving the core error. This is an explicitly fictional composite, not a customer result.

TL;DR

An evaluator cannot rescue tests built from the same mistaken assumption as the code; it needs requirement-derived examples, external invariants, adversarial cases, or accountable human decisions that exist outside the generated solution. Inspect shared-assumption risk, external test oracle, adversarial evidence, and unresolved-decision route on one recent change. If another qualified reviewer cannot reach the same decision from the stored evidence, pause and repair the earliest missing handoff before increasing automation or output. The goal is dependable delivery that the team can explain, challenge, and maintain.

In this article

1. Consistency can hide shared error
2. Evaluator quality has two failure directions
3. Independent evidence must enter the loop
4. Stop repeated correction without information gain
5. Four checks for a real repository change
6. Frequently Asked Questions

Consistency can hide shared error

Risk of coding before testing, July 2026 supports the evidence boundary. A recent preprint in a bounded setup that needs replication.

Use the table as a diagnostic, not benchmark data. Replace illustrative entries with repository evidence before changing staffing, policy, or release authority.

Evaluator quality has two failure directions

Code reviewers can falsely accept unsafe code or falsely reject correct code. The 2026 reliability study showed prompt-sensitive trade-offs and that filters shifted them. Track both directions. An evaluator that blocks everything is not safe; it has moved cost into false rejection.

Sequence matters. When an early input is missing, later activity can look complete while a reviewer inherits the unresolved uncertainty.

Independent evidence must enter the loop

Bring in behavior examples written from the approved specification, registry data, schema checks, mutations, specialist findings, and human product decisions. Label whether evidence is deterministic, model-judged, or human. Independence is a property to inspect, not a model count.

Reliability of LLM code reviewers, 2026 provides a second check. Small benchmark functions and prompt-sensitive trade-offs; execution did not prove correctness.

Mistake: Assuming a second AI grader is automatically independent. Fix: Inspect shared prompts context criteria and oracles, then inject evidence from a genuinely different source.

The fix is procedural because a procedure can be rehearsed and audited. A warning without ownership rarely survives a busy release week.

Stop repeated correction without information gain

Stop after repeated failures with no new evidence, specification mismatch, unchanged surviving mutations, budget limits, or sensitive uncertainty. A fourth stylistic rewrite is not correction if the core premise remains untested.

paqad-ai v1.67.0 was the current public release when this article was verified on July 21, 2026. The repository describes local workflows, risk routing, structured state, checks, documentation sync, and audit records. Those are inspectable mechanisms, not independent proof of a business result.

Use the AI workflow audit guide for current-state mapping, compare the evidence bar with production-ready AI code, and read the AI coding consultant guide when outside support is relevant.

Decision rule: Allow another loop turn only when it introduces new evidence or a materially different test of the disputed behavior.

Run the evaluator-independence audit in a 45-minute working review with the product owner, module owner, and final reviewer. Do not turn it into a score. Read the decision rule aloud: Allow another loop turn only when it introduces new evidence or a materially different test of the disputed behavior. Record disagreements, missing evidence, and the person responsible for the next action. Repeat the review after one real feature so the team can compare whether the artifact reduced ambiguity or only added paperwork.

Four checks for a real repository change

Apply the article’s argument to one active change. The point is to discover whether the process works under normal delivery pressure, not whether the team can describe an ideal process.

• Shared-assumption risk: identify the exact artefact and the person who may change it.
• External test oracle: ask a second reviewer to use it without oral context.
• Adversarial evidence: bind the result to the code revision or decision it supports.
• Unresolved-decision route: define the continue, pause, reject, and recovery choices.

Record every place the reviewer must guess. A guess about wording may require better documentation. A guess about a deterministic result may require an executable check. A guess about risk or product intent requires an owner, not another automated opinion.

Fix one gap and run the same review again. The improvement is credible when the next person can reach a defensible decision from the artefacts, and when an unresolved result remains visible as unresolved. That is a stronger outcome than a faster first answer.

A usable review record for this decision should identify the accepted request, the exact revision being judged, the source of shared-assumption risk, and the owner responsible for unresolved-decision route. It should also preserve disagreement: a reviewer who finds weak external test oracle or missing adversarial evidence needs a visible way to return the work as incomplete. That record gives the next developer a reliable starting point and prevents a confident summary from becoming the only explanation of what happened.

An evaluator should be allowed to say that no trustworthy oracle exists. That verdict reveals a specification problem rather than an evaluation failure. Route it to the product or domain owner, capture the clarified behavior, and add an independent example before restarting the loop. Asking the model to invent another test only makes the shared assumption more consistent. The useful outcome is a stronger target that future workers and evaluators can both use, not a green result obtained by increasing agreement among artifacts derived from the same uncertain premise.

For a related view of the delivery system, read how AI-generated code changes review and how a reviewable AI feature plan works.

If your team is generating work faster than it can verify and own it, explore the AI Delivery Programme.

Frequently Asked Questions