TL;DR
Deterministic proof records a reproducible result from a defined rule or execution, while model judgment offers a reasoned but challengeable interpretation; both are useful only when they are labelled and routed correctly. Inspect repeatable check result, recorded inputs, model interpretation, and decision owner on one recent change. If another qualified reviewer cannot reach the same decision from the stored evidence, pause and repair the earliest missing handoff before increasing automation or output. The goal is dependable delivery that the team can explain, challenge, and maintain.
In this article
Last updated October 14, 2026
7 min read
Deterministic proof records a reproducible result from a defined rule or execution, while model judgment offers a reasoned but challengeable interpretation; both are useful only when they are labelled and routed correctly. This question matters because faster generation moves work into decisions that still require evidence and ownership.
AI development reports often make every successful result look equal.
Tests passed. Architecture passed. Security passed. Specification passed. Ready to merge.
The uniform green presentation hides different kinds of knowledge.
A test runner can prove that a command exited successfully in a recorded environment.
A model can judge that an architecture looks coherent.
A human can decide that the available evidence is sufficient for production.
These claims support one another. They are not interchangeable.
Four useful evidence categories
Deterministic evidence
The same defined input and method should produce a checkable result.
Examples:
Evidence categories
- lint, type, test, and build commands;
- schema validation;
- file existence;
- package registry query;
- dependency-direction rule;
- hash comparison;
- static pattern check.
Record the command or method, environment, time, exit code, and output reference.
Deterministic does not mean complete. A passing suite proves only what the suite and environment cover.
Measured evidence
Measurements depend on method and conditions.
Examples:
- response latency;
- memory use;
- coverage;
- mutation score;
- complexity;
- accessibility or performance audit.
Record the instrument, environment, sample, baseline, and limitation. A number without method is decoration.
Model-judged evidence
Some questions require interpretation:
- Does the plan fit the architecture?
- Does a test meaningfully cover an acceptance criterion?
- Is the design consistent?
- Which threat is likely?
- Does documentation explain the changed behaviour?
A model can perform useful specialist analysis. Record its inputs, reasoning, uncertainty, and claim boundary. Do not present the conclusion as a deterministic pass.
Human-decided evidence
Some outcomes are choices rather than discovered facts:
- which product behaviour should apply;
- whether to accept a risk;
- whether an architecture exception is justified;
- whether the available evidence supports production.
Record the authorised owner, alternatives, evidence considered, reason, and time.
Add a fifth state: unknown
Use it when:
- evidence is missing;
- sources conflict;
- a check did not run;
- the evaluator lacks capability;
- the decision owner is unavailable;
- the claim extends beyond the measurement.
Unknown should route to more evidence, a human decision, re-planning, or a stop. It should not be converted to green by a summary.
Build reports around claims
For every material conclusion, record:
Compare the decision
| Field | Question |
|---|---|
| Claim | What exactly is being asserted? |
| Category | Deterministic, measured, model-judged, human-decided, or unknown? |
| Source | Which artifact, command, or person supports it? |
| Scope | Which files, environment, behaviour, or period does it cover? |
| Limit | What does it not establish? |
| Status | Pass, fail, inconclusive, or decision required? |
This makes the report inspectable rather than merely reassuring.
Examples of honest boundaries
“All tests passed in CI” does not prove every acceptance criterion has a meaningful test.
“The registry contains the package” does not prove the package is legitimate or suitable.
“No high-severity issue was found by the model reviewer” does not prove the code is secure.
“The reviewed hash matches” proves artifact identity, not content correctness.
“The human approved” does not prove meaningful oversight unless the person had evidence and authority.
A failure walkthrough makes the hidden cost visible
Imagine the change reaches review with a confident summary, plausible code, and a green check. The reviewer then discovers that repeatable check result was never agreed. That discovery changes the value of every later artefact because the team may have verified the wrong outcome.
Work backward from that point. Ask who should have established repeatable check result, where recorded inputs should have been visible, how model interpretation would have exposed the mismatch, and who had authority over decision owner. This sequence shows whether the failure came from the model, the task, the repository context, or the delivery system around them.
The repair must move the discovery earlier. Put an objective check in automation when the condition is deterministic. Use a named human decision when consequences or trade-offs require judgment. Preserve the result beside the change so the next reviewer does not repeat the investigation.
The walkthrough is complete only when the team can state a stop condition. If the necessary evidence is absent or contradictory, the workflow pauses instead of asking the agent to make the uncertainty disappear through another attempt.
A usable review record for this decision should identify the accepted request, the exact revision being judged, the source of repeatable check result, and the owner responsible for decision owner. It should also preserve disagreement: a reviewer who finds weak recorded inputs or missing model interpretation needs a visible way to return the work as incomplete. That record gives the next developer a reliable starting point and prevents a confident summary from becoming the only explanation of what happened.
The first rollout should stay narrow. Choose a change with a known owner and a result the team can observe, then compare the planned evidence with what the workflow actually produces. Keep the exception path explicit. If the evidence is incomplete, record that outcome honestly instead of converting uncertainty into a passing verdict.
Review the record after the change is accepted. Ask whether it still explains why the team trusted repeatable check result, how recorded inputs affected the decision, which result established model interpretation, and who remained accountable for decision owner. If those answers disappear after the meeting or agent session ends, the workflow has stored an outcome without preserving the reasoning needed to maintain it. Correct that gap before using the same approach on work with a wider impact.
Use the result to change one operating rule, not to produce another advisory document. The rule might require a stored artefact, a deterministic check, a specialist review, or a decision pause. Give it an owner and test it on the next comparable change. If the same ambiguity returns, the correction did not reach the point where work actually moves.
Design the review report so readers cannot confuse evidence classes. Put the command, revision, environment, and output beside deterministic results. Put method and conditions beside measurements. Label model findings as interpretations with supporting references and unresolved uncertainty. Record human decisions with the owner and consequence accepted. This structure lets the forms support one another without changing their meaning. A reviewer can use a model finding to choose a deterministic check, or use a failed check to request judgment, while still seeing exactly what was proved and what was decided.
For a related view of the delivery system, read how AI-generated code changes review and how a reviewable AI feature plan works.
Sources used
- NIST AI RMF Playbook, Measure, voluntary measurement and oversight guidance.
If your team is generating work faster than it can verify and own it, explore the AI Delivery Programme.
Frequently Asked Questions
What is the difference between deterministic proof and model judgment in AI code review?
Deterministic proof records a reproducible result from a defined rule or execution, while model judgment offers a reasoned but challengeable interpretation; both are useful only when they are labelled and routed correctly. Judge the practice on one real change and keep the supporting evidence beside the work.
Which software checks require repeatable evidence instead of an AI opinion?
Track repeatable check result, recorded inputs, model interpretation, and decision owner together. A single speed or output number cannot show whether the team can safely accept and maintain the result.
Which checks should always be deterministic?
Builds, types, schemas, package resolution, file presence, policy rules, and repeatable tests should produce recorded execution results whenever the repository supports them.
Where is model judgment useful in code review?
Models can identify likely gaps, explain architecture, compare patterns, and propose risks. Those findings should invite verification rather than masquerade as executed proof.
Why must the report label evidence types clearly?
A reader needs to know which result can be reproduced, which was measured under conditions, which is an interpretation, and which was decided by a human owner.
Can several model judgments become deterministic proof?
No. Agreement may increase confidence but does not turn reasoning into execution. A deterministic claim still needs a defined rule and recorded result.
