TL;DR
A checkpoint should tell you what was checked, on which code, and whether the required work actually ran. Define those rules before the agent starts. Save the command, environment and result with the change, including skipped checks and missing evidence. A successful command alone may not mean the required test passed. If a required check cannot run, pause the dependent action and show what is missing. Keep the final claim as narrow as the evidence supports.
In this article
Last updated October 15, 2026
6 min read
A test runner can finish successfully while the behaviour you care about remains unchecked. That is not always a bug in the runner. It may be doing exactly what its own rules allow.
The checkpoint contract is the team’s agreement about what counts as enough evidence to continue. It connects the runner’s result to a specific requirement. Without that agreement, an agent can accurately report the exit code while overstating what the change has proved.
One skipped test can still produce exit zero
Four small test fixtures were executed for this article with Python 3.14.2 and its standard unittest runner. Each ran in a separate directory using the same discovery command. The passing case checks that two plus two equals four; the failing case deliberately expects five. The skipped case marks that same test unavailable. The empty case contains no test.
Observed runner results
| Fixture | Exit | Output |
|---|---|---|
| One passing test | 0 | OK |
| One failed test | 1 | One failure |
| No tests | 5 | No tests ran |
| Only test skipped | 0 | One skipped |
These are constructed runner examples, not observations of an AI agent or a production defect. The useful result is the last row: exit zero did not establish that the required assertion had executed. The output disclosed the skip, so a checkpoint that reads only the exit code would discard relevant evidence.
The Python unittest documentation describes skipped tests as a distinct outcome. Another runner may use different exit rules. For example, pytest documents exit code 5 for no collected tests. Do not invent a universal mapping such as “exit 2 means contract error”; use the semantics of the actual command and version.
Say what must run, not just which command to launch
For the example, suppose the requirement is that test_required must execute and pass. The contract should reject both the empty and skipped cases as insufficient evidence for that requirement. The failed assertion is different: the check ran and found a mismatch.
For a repository change, choose the required test identifiers or behaviours before running the check. “At least one test ran” is too weak if the one that ran concerns a different feature. If discovery changes, review the required set rather than silently lowering the condition until the run passes.
This contract is a design example, not an installed enforcement mechanism. A wrapper or CI rule must actually evaluate the result if the team wants automatic blocking. The related article on testing from the specification explains how to decide what the test should establish in the first place.
Bind the result to what actually ran
Record the source revision, working directory, command arguments and relevant tool versions. A commit identifier alone is insufficient if uncommitted files changed the input. Use a clean checkout or preserve a record of the actual tree being tested, including any relevant generated files or configuration.
Give each run its own evidence location. A report left by yesterday’s successful run should not satisfy today’s command when that command never produced a report. Record when the run started and ended, how the report was obtained and which input it describes. Store configuration names and safe fingerprints without copying secrets into the evidence.
When available, use a documented machine-readable test report for test identities and outcomes. Retain enough raw output to diagnose disagreements with the parsed result. A file existing on disk is not sufficient: the checkpoint needs to establish that it is complete, valid and associated with this run.
If the command times out, the tool is missing or the report cannot be parsed, record that execution problem. Do not label the feature correct or incorrect merely because the verification environment could not answer. The guide to honest AI evidence verdicts develops that distinction between a failed behaviour and an unresolved check.
Check the checkpoint’s own failure paths
Before relying on a new contract, exercise it with a passing result, a failed assertion, a skipped required test and an empty selection. Also try a missing report, an old report and a result associated with another revision. Define the expected handling first. These additional contract checks are proposed here; the executed demonstration covered only the four runner fixtures above.
A timeout or a malformed report should lead to an actionable explanation: what could not be established, where the evidence is, and who can repair the environment or change the requirement. Keep the failed attempt even if a later attempt succeeds. Otherwise a passing retry can conceal an intermittent problem that the team needs to understand.
Version the contract when its required work or result interpretation changes. A reviewer should be able to see whether the code improved, the environment changed, or the checkpoint simply became easier to satisfy. Keep that explanation with the change instead of replacing it with one undifferentiated green status.
If your team needs help making automated checks useful to reviewers, request an AI Delivery Programme conversation.
Frequently Asked Questions
How do you design a deterministic checkpoint for AI-generated code?
Choose an acceptance rule a tool can evaluate consistently, and keep decisions requiring judgment outside that rule. Start with a specific claim, such as a named regression test passing on the proposed revision. A checkpoint called “quality passed” is difficult to challenge because its promise is undefined.
What must a checkpoint contract record to be reproducible?
Preserve enough information for another person to reconstruct the relevant inputs and invocation. Include dependencies on fixtures, services and configuration that are not obvious from the command. Reproducibility may remain limited when an external service changes; state that dependency instead of promising identical results.
Can a flaky test ever be a required checkpoint?
It can, but a pass after repeated attempts needs an explicit policy. Investigate whether variation comes from the product, test or environment. If the team temporarily removes the check from blocking, record the gap, its owner and the condition for restoring it. Do not quietly treat retries as repairs.
What if some skipped tests are intentional?
Distinguish optional checks from required ones before execution. A platform-specific test may be irrelevant on one platform and essential on another. Record that applicability rule; a blanket ban on every skip can be as misleading as ignoring all skips.
Can evidence be reused after a documentation-only change?
Only if the contract establishes that the changed material cannot affect what the evidence claims. Documentation may include executable examples, configuration or generated inputs. Review the dependency relationship before reusing a result, rather than assuming every file ending in Markdown is harmless to verification.
Who should be allowed to change the checkpoint while fixing a failure?
Assign that authority separately from the ability to edit implementation code. Sometimes the check is wrong, but weakening it changes the acceptance rule. Make the proposed change visible to the requirement’s owner or reviewer and explain why the previous rule no longer represents the intended behaviour.
