Last updated September 18, 2026
4 min read
A fictional refund test asserts HTTP 200 and one database write. The implementation refunds the wrong amount. The test passes because it proves activity, not the acceptance criterion. Green output hides a weak oracle. This is an explicitly fictional composite, not a customer result.
In this article
Test presence is not test strength
Count assertions only after checking what they assert. A test that confirms a response exists or a mock was called may leave the business result untouched. Begin with the criterion and write the smallest failure that would violate it.
Risk of coding before testing, July 2026 supports the evidence boundary. A recent preprint in a bounded setup that needs replication.
| Item | Inspectable value |
|---|---|
| 1 | Criterion AC-04 |
| 2 | HTTP 200 test |
| 3 | 1 amount mutation |
| 4 | 0 detected behavior failures |
Use the table as a diagnostic, not benchmark data. Replace illustrative entries with repository evidence before changing staffing, policy, or release authority.
Trace every promise to observable evidence
Give each criterion a stable ID and list the test, assertion, fixture, and expected failure it maps to. Flag promised behavior with no test and tests with no requested purpose. Traceability keeps the review on outcomes instead of file count.
Sequence matters. When an early input is missing, later activity can look complete while a reviewer inherits the unresolved uncertainty.
Mutation reveals decorative assertions
Mutation testing changes operators, conditions, and returned values to see whether tests notice. If changing refund amount calculation leaves the suite green, the oracle is weak. Mutation score is not correctness, but it gives stronger evidence than a passing suite alone.
AI-generated test oracles, ASE 2025 provides a second check. Balanced benchmark evidence in a Java corpus, not a correctness guarantee.
The fix is procedural because a procedure can be rehearsed and audited. A warning without ownership rarely survives a busy release week.
Use independent examples before implementation anchors them
The July 2026 preprint found that faulty code shown before test generation reduced fault detection in its setup. ASE 2025 benchmark evidence also shows AI-generated oracles can approach human average mutation performance in one corpus. Keep both findings: generated tests can help, and independence still matters.
paqad-ai v1.67.0 was the current public release when this article was verified on July 21, 2026. The repository describes local workflows, risk routing, structured state, checks, documentation sync, and audit records. Those are inspectable mechanisms, not independent proof of a business result.
Use the AI workflow audit guide for current-state mapping, compare the evidence bar with production-ready AI code, and read the AI coding consultant guide when outside support is relevant.
Decision rule: Do not accept a green suite until at least one deliberate break in each critical behavior makes the mapped test fail.
Run the promise-to-test traceability sheet in a 45-minute working review with the product owner, module owner, and final reviewer. Do not turn it into a score. Read the decision rule aloud: Do not accept a green suite until at least one deliberate break in each critical behavior makes the mapped test fail. Record disagreements, missing evidence, and the person responsible for the next action. Repeat the review after one real feature so the team can compare whether the artifact reduced ambiguity or only added paperwork.
Frequently Asked Questions
What is a test oracle?
It is the mechanism that decides whether observed behavior is correct. An assertion, expected event, snapshot, or property can act as an oracle. Its value depends on what failures it can detect.
Do all teams need mutation testing?
Use it where changed behavior is important and test strength is uncertain. Start on the changed scope. Large legacy suites may need gradual adoption to keep runtime and noise manageable.
Can AI write useful tests?
Yes. Benchmark evidence shows strong results are possible in bounded settings. Preserve independent intent, review assertions, and test the tests with mutations or deliberately broken behavior.
What next?
Apply the promise-to-test traceability sheet to one active feature. If the repository cannot produce the required state and evidence, install paqad-ai and test the workflow locally with a named human decision owner.