TL;DR
Build and test the core path once before you automate repetition. Define the task, expected result, evaluator, stop condition and recovery path, then run one real change from start to finish with a human watching the handoffs. If the team cannot explain why a single pass succeeded or failed, a faster loop will repeat uncertainty rather than remove it.
In this article
- Rehearse one complete change before adding recurrence
- Core 1: an owned goal
- Core 2: a reviewable specification
- Core 3: current, task-relevant context
- Core 4: an independent evaluator
- Core 5: durable state
- Core 6: stop, budget, and recovery rules
- Core 7: human authority
- Core 8: evidence and feedback
- Where paqad-ai fits
- Source notes
- Frequently Asked Questions
7 min read
These two reader concerns frame the article: What should you build before automating an AI coding loop? and Should we build the workflow first or let the agent iterate and add controls later?. The links show the wording and context of the questions; the answer below uses primary or official sources for material claims.
Weak signal and stronger evidence
| Weak signal | Stronger evidence |
|---|---|
| The agent or tool says the work is complete | The named result is checked against the exact revision |
| A policy exists in a document | The rule is visible in the working delivery path |
| More output is produced | Accepted outcomes improve without hidden review debt |
Those pieces make a loop run.
They do not make the loop worth repeating.
The quality of the core determines whether each cycle compounds value, debt, or uncertainty.
Anthropic’s agent-design guidance recommends starting with the simplest solution that meets the need and adding complexity only when it improves outcomes. It also describes environmental feedback, human checkpoints and explicit stopping conditions as parts of an agent loop. That is vendor guidance rather than independent proof, but it gives teams a useful engineering sequence: make one path observable, verify it, and only then automate its repetition.
Rehearse one complete change before adding recurrence
Consider an explicitly illustrative checkout change: allow a signed-in customer to recover from an expired payment session without creating a duplicate order. Before a loop retries implementation and evaluation, run the change once from intake to verdict.
Single-pass rehearsal
- Approve the behaviour. Define the customer outcome, the duplicate-order invariant and the cases that remain outside this change.
- Prepare current context. Identify the checkout owner, payment-session code, existing recovery pattern, commands and affected documentation.
- Make one bounded change. Limit the diff and record every unexpected file or dependency the implementation needs.
- Evaluate the outcome. Run the deterministic checks, inspect the failure path and ask an independent reviewer whether the behaviour matches the approved requirement.
- Exercise the stop. Introduce a case the specification does not resolve and confirm that work pauses with a useful question instead of guessing.
- Preserve the receipt. Keep the requirement version, revision, results, decision, remaining gap and next action.
This rehearsal exposes a weak foundation without multiplying it. If the evaluator rewards a test that merely repeats the implementation, fix the evaluator. If the context points to an obsolete payment pattern, fix retrieval. If nobody owns the duplicate-order decision, resolve ownership. Recurrence becomes useful only after the team can locate the reason for each pass, failure and pause.
Anthropic’s agent-evaluation guide distinguishes a task from a trial and the final outcome from the execution trace. It recommends choosing code-based, model-based and human graders for the parts each can judge. For this checkout example, a deterministic test can verify that one order exists, a trace check can show whether the agent crossed scope, and a person can judge whether the recovery behaviour matches the product promise.
What to prove before repeating the path
| Foundation question | Evidence from one pass | Reason to delay automation |
|---|---|---|
| Is the outcome settled? | Approved behaviour and invariant | Product owner still needs to choose |
| Can the result be checked? | Independent outcome and failure checks | Evaluator repeats the maker’s assumption |
| Can the run stop usefully? | A real pause returns evidence and options | The agent guesses or retries unchanged work |
| Can another person resume? | Current state and next action survive | Progress exists only in chat context |
| Can the change be reversed? | Tested rollback or bounded recovery | Side effects cannot be identified or contained |
Core 1: an owned goal
The loop needs an outcome that a person or team owns.
Weak goal:
“Improve checkout.”
Stronger goal:
“Allow a signed-in customer to recover from an expired payment session without creating a duplicate order.”
The stronger goal names observable behaviour. It still needs product confirmation and evidence.
If the goal remains vague, do not use more cycles to discover product intent through code.
Core 2: a reviewable specification
Record:
- acceptance criteria;
- edge cases;
- invariants;
- scope;
- sensitive boundaries;
- proof method;
- unresolved decisions.
Use a concise specification for small work and a frozen, versioned specification for wide or sensitive work.
When the target changes, record the change.
Core 3: current, task-relevant context
The loop does not need the maximum possible context.
It needs:
- owning module;
- relevant rules;
- closest existing patterns;
- live commands;
- affected documentation;
- source authority;
- drift warnings;
- retrieval paths for deeper inspection.
Treat retrieved or compressed context as a lead to verify, not unquestioned truth.
Core 4: an independent evaluator
The evaluator should compare work with the specification, not the maker’s confidence.
Layer:
- artifact validity;
- deterministic checks;
- traceability;
- specialist judgement;
- human verdict.
Allow pass, fail, inconclusive, decision required, re-plan required, and budget exhausted.
A binary green light turns missing evidence into pressure for false certainty.
Core 5: durable state
Store:
- workflow and active step;
- approved artifacts;
- results;
- decisions;
- failures;
- next action.
Separate short-lived operational state from long-lived engineering precedent. Add scope and invalidation to memory.
Core 6: stop, budget, and recovery rules
Define:
- success;
- deterministic failure;
- uncertainty;
- decision boundary;
- token, time, retry, diff, and review budgets;
- safety stops;
- rollback and re-planning.
Persistence without these controls is repeated motion.
Core 7: human authority
The human does not need to approve every tool call.
They need:
- a specific decision;
- evidence and limitations;
- realistic options;
- timing before commitment;
- authority to approve, reject, redirect, narrow, or request more evidence;
- a durable reason.
The agent runs the inner cycle. The team owns the outer loop.
Core 8: evidence and feedback
The final receipt should connect:
- target;
- changed scope;
- deterministic results;
- measurements;
- model-judged findings;
- human decisions;
- documentation;
- artifact identity;
- uncertainty;
- verdict.
Then use review findings, incidents, overrides, and rework to improve the workflow itself.
Keep that receipt outside the agent’s conversation so another person can inspect it later. Eliyce’s delivery record shows the kind of revision-linked evidence a team can retain across planning, implementation, checks and human decisions.
Loop Engineering includes engineering the loop.
Where paqad-ai fits
paqad-ai is designed as the orchestration around this core: routing, planning, specification, roles, scoped development, review, checks, documentation, decisions, state, evidence, and provider adapters.
It is not one skill and not one agent.
It also cannot supply product judgement the team has not made or guarantee that the loop’s specification and checks are sufficient.
The mature promise is narrower and more useful:
paqad-ai gives teams a structured place to encode the core, run the workflow, and inspect the evidence.
Do not copy the recurrence and skip the foundation.
A fast loop amplifies what is already inside it.
Source notes
- Anthropic, Building effective agents.
- Anthropic, Demystifying evals for AI agents.
- Addy Osmani, Loop Engineering.
- Addy Osmani, Own the Outer Loop.
- Product mechanism: first-party details from What is Paqad? and the current Eliyce/paqad-ai repository.
Decision rule
Proceed only when the team can name the owner, inspect current evidence and stop or reverse the work without reconstructing an agent conversation.
Paqad is a first-party Eliyce product. Teams that want to test these controls on their own repository can explore the AI Delivery Programme.
Frequently Asked Questions
What should you build before automating an AI coding loop?
Use the body’s decision process first. For a reusable team rule, record the boundary and accountable owner in repository-controlled files, attach proof to the exact revision, and review the decision whenever the tool, permissions or task class changes.
Should we build the workflow first or let the agent iterate and add controls later?
Start with one bounded task and compare accepted outcomes, correction work and review demand. A tool capability is useful only when the surrounding team can verify and own the result.
How do you know the evaluator is independent enough?
Change one assumption in the implementation and check whether the evaluator notices. Seed a forbidden side effect, stale document or test that confirms the wrong behaviour. An evaluator earns trust by rejecting known-bad cases for the right reason, not by agreeing with a clean run.
What should happen when the loop reaches the same failure twice?
Compare the attempted actions and evidence. If the second attempt repeats the same diagnosis or edit, stop automatic retries and require a new plan or human decision. A different error after measurable progress may justify another bounded attempt; an unchanged failure usually signals missing context or a bad strategy.
Which state must survive if a human pauses the loop?
Keep the approved goal and specification version, completed step, exact revision, tool results, decisions, unresolved question and permitted next actions. The person should be able to resume, redirect or reject the work without reading the full transcript or asking the model to remember it.
When is a fixed workflow better than an autonomous loop?
Use a fixed workflow when the steps and branches are known, the same checks apply each time and predictability matters more than model-directed exploration. Reserve an autonomous loop for work whose next action cannot be enumerated in advance and whose environment provides reliable feedback at each step.
