TL;DR
A passing test tells you little unless it would notice the mistake you care about. Start with an expected result from the requirement, then run the test against a version with that behaviour deliberately broken. Check why it fails: a wrong result is useful evidence; a missing dependency is not. Run the same test against the corrected code. For broader checks, try other small changes and investigate any that survive. This tests the strength of the assertions without claiming the whole system is correct.
In this article
Last updated October 15, 2026
6 min read
A test can execute a calculation and still miss the fact that its answer is wrong. Checking that the result is a number does not check whether it is the number the user needs.
This is the practical problem behind a developer question about generated tests passing on old code. The author describes rerunning new tests without the implementation change. That can expose a weak regression test, but it needs interpretation: a test for unchanged behaviour should often pass before and after a refactor.
For a bug fix, identify the particular mistake the new test must detect. The review then has a concrete question: does this assertion distinguish the broken behaviour from the intended result?
Start with a result you can explain without reading the implementation
The expected result might come from an agreed product rule, a published interface contract or an independently checked example. Testing terminology calls the rule used to judge the result an oracle. It need not be complicated. For a known input, it can be a single expected value with a clear reason behind it.
A test that calculates its expected value by repeating the implementation’s formula can repeat the same defect. Ask the reviewer to explain the expected answer from the requirement instead. If two reasonable answers remain possible, settle that ambiguity before asking an agent to generate more assertions.
For this article, a small Python experiment used an explicit rule: remaining capacity equals the limit minus the amount used, with zero remaining when usage reaches or exceeds the limit. Inputs were nonnegative integers. This is a constructed demonstration, not a customer incident or a comparison of AI models.
What the executed tests actually detected
Two test suites ran against six versions of the function. The weaker suite checked only that each result was an integer. The stronger suite checked four expected results: with a limit of 10, usage of 0, 7, 10 and 13 must leave 10, 3, 0 and 0 respectively.
The versions changed the calculation while keeping the test inputs fixed.
The absolute-difference version is particularly revealing. When usage is 13 and the limit is 10, it returns 3 instead of 0. That is still an integer, so the type check accepts it. The value assertion fails with the specific mismatch 3 != 0. Restoring the correct calculation makes the same test pass.
The always-zero version shows why testing only the exhausted case would also be weak. Zero is correct for usage of 10 or 13, but wrong when capacity remains. The selected inputs cover both sides of the rule and its boundary. They do not cover every possible input type or integration with a real system.
The experiment retained the source variants, test files, commands, output and hashes. It used Python’s standard test runner, not a mutation-testing product. Its result demonstrates these assertions’ sensitivity to selected changes; it does not measure how often an AI system generates good tests.
Read the failure before calling the test useful
The missing-import version returned a failing exit code in both suites. However, the runner could not load the function, so neither suite established whether the calculation was correct. Counting that as successful bug detection would confuse a setup problem with evidence about behaviour.
Use a separate, disposable copy of the code for deliberate changes. Keep them out of the release candidate. Run the relevant tests on the known-good version first, apply one change, inspect the failure, then restore the original and rerun. A test that already fails before the experiment cannot provide the same comparison.
For an existing defect, use a revision that reproduces it when practical. For a new feature, deliberately violate a requirement the test claims to protect. Avoid changing the assertion to match the broken code merely to get the run back to green.
A surviving change is a question to investigate
Mutation testing automates small changes to code and observes whether tests detect them. A separate developer question about mutation testing asks what this says about the original program. The useful distinction is between finding weaknesses in tests and proving that the original implementation has no defects.
In our experiment, replacing the correct calculation with an equivalent conditional left both suites passing. For the declared integer inputs, the two branches preserve the rule: subtract while usage is below the limit, otherwise return zero. That surviving change does not require an assertion against the spelling of the implementation.
PIT’s documentation explains equivalent mutations and separates outcomes such as detected changes, survivors, uncovered code and run errors. Those categories help avoid treating every survivor as a missing test or every failed execution as successful detection.
Investigate a survivor in this order: did the test exercise the affected code, did the change alter relevant behaviour, and did the assertion observe that behaviour? If the change is equivalent, record the reasoning. If it changes something the product promises, improve the example or assertion. An overall mutation score cannot make that decision for you.
Generated assertions can help, with a defined evidence boundary
The authors’ summary of an ASE 2025 study reports similar average mutation scores for generated and human-designed test oracles across 135 Java projects. It also identifies limitations in generating complex oracles. That is evidence about the evaluated assertions and dataset, not a guarantee for another model, language or repository.
Review generated tests using the same behavioural standard as other tests. A useful record names the requirement, input, expected result, deliberately broken version, observed failure and corrected result. It should also state what remains unchecked. More generated cases are valuable when they explore missing behaviour, not when they repeat the same assertion with different names.
For deciding the expected result before implementation, see writing tests from the specification. For separating test execution from the final acceptance decision, read designing a loop evaluator.
If your team needs to make this review part of everyday AI-assisted development, request an AI Delivery Programme conversation.
Frequently Asked Questions
How do you verify that AI-generated tests would catch the bug?
Ask which incorrect outcome the test is intended to reject. Then inspect whether the test reaches the relevant code and observes that outcome. A convincing demonstration includes the cause of failure, not just its colour or exit code. Keep the test unchanged while comparing the defective and corrected versions.
What proves a generated test is stronger than a green test run?
Evidence that it distinguishes relevant alternatives gives you a narrower, defensible claim. It may detect an incorrect boundary value while leaving concurrency or persistence untested. State that scope explicitly so the next reviewer knows which risks still need another kind of check.
Should tests for a refactor fail on the previous version?
Usually, tests of preserved behaviour should pass on both versions. Requiring them to fail on the old code can reward assertions about implementation details. Review the refactor against the behaviour it must preserve, and use a separate deliberate defect to assess a test’s sensitivity when that would be useful.
What if the test fails only occasionally?
Control relevant sources of variation, such as clocks, randomness or shared state, before interpreting the result. Compare repeated runs of the unchanged version as well as the deliberately altered version. If failures occur in both without a clear cause, the experiment has not isolated the defect.
What if mutation testing takes too long for every change?
Start with the changed logic and the behaviours with the greatest consequences. Broader runs can happen separately when appropriate for the repository. Record which areas were excluded; a targeted result should not be presented as evidence about the whole codebase. Timing constraints change scope, not what the result proves.
What if nobody agrees on the expected result?
Bring the disputed example to the person responsible for the product rule or interface contract. Record the decision and its limits before turning it into an assertion. Asking another model to choose an answer may reveal alternatives, but it does not resolve who has authority to define the behaviour.
