Why should teams design tests before AI writes code?

TL;DR

Before asking AI to implement a feature, write down what a user should get and a few examples that distinguish correct behaviour from a plausible mistake. Use those examples to judge the generated code. Otherwise, tests written from the code may repeat its misunderstanding and still pass. You do not need to design every class first. Start with one clear rule, check the expected answers yourself, and keep unresolved product questions visible.

In this article

1. Decide what the result means before writing the assertion
2. A small demonstration of the wrong answer passing
3. Give the test writer a useful starting point
4. Avoid turning test-first work into a complete design up front
5. Frequently Asked Questions

Last updated October 15, 2026

5 min read

A project list should show the first three active projects. The implementation takes the first three projects, then removes inactive ones. A test copied from that implementation can pass while the page shows fewer results than requested.

The important question is where the test gets its expected answer. A specification can supply that answer, provided it says enough to distinguish the two behaviours.

Decide what the result means before writing the assertion

Consider this illustrative requirement: preserve the existing order of projects, ignore inactive projects, and return up to three active project names. If fewer than three are active, return all of them. An empty input returns an empty result.

This leaves the developer free to choose a loop, a query or a collection operation. It does settle what a user should receive.

Write the expected names directly in the test. Calculating them by calling the same function being tested would only compare the function with itself. A separate helper can have the same weakness if it repeats the implementation’s algorithm without an independent reason for trusting it.

A small demonstration of the wrong answer passing

The following Python functions make the ordering mistake visible. Each input item contains a name and a Boolean active flag. Both versions preserve input order.

# Wrong: restrict the records before selecting active projects.
def wrong(rows):
    return [name for name, active in rows[:3] if active]

# Matches this example's requirement.
def intended(rows):
    return [name for name, active in rows if active][:3]

A local execution of this synthetic example produced these results. The expected answers were written explicitly from the stated rule.

The wrong function passed three of five cases. When the mixed-case expected answer was instead copied from its output, A, C, that assertion accepted the wrong function and rejected the corrected one.

This demonstrates a test-design failure, not how often an AI model makes it. No model or application was tested here. A real project list would also need checks for its database query, permissions, pagination and user interface.

A July 2026 preprint on coding before testing investigates the related risk in model-generated code and tests. Its authors report that faulty implementations can influence later test assertions. The study selected faulty Python benchmark solutions, excluded execution failures and favoured faults that fewer reference tests exposed. Its results therefore describe a selected set of mistakes, not the defect rate of an everyday codebase. The authors also caution that larger systems, human oversight and different workflows may behave differently. Use the finding to question a test’s expected answer, not to treat test-first work as a correctness guarantee.

For features that themselves produce AI output, Anthropic’s evaluation guidance distinguishes checks performed by code, models and people. Choose the check to fit the promise: an exact field can be asserted directly; an explanation’s usefulness needs judgment. The FAQ below addresses how to handle that difference without requiring one fixed sentence.

Give the test writer a useful starting point

For a new feature, give the person or agent designing acceptance checks the request, agreed examples, important constraints and open questions. Let them propose the expected outcomes before using the implementation as an explanation of what should happen.

They may still need code access to connect a test to the application. Method names, fixtures and setup routines help make a test executable. They should not silently redefine the result that counts as correct.

For the project list, an unanswered question might be whether pinned projects take priority over the existing order. The developer can investigate how pinning works, but a convenient implementation cannot decide that product rule by itself. The reviewable feature plan is where that decision can stay attached to the change.

Avoid turning test-first work into a complete design up front

Developers ask both whether writing tests before implementation helps in practice and how much class design must be specified first. These discussions identify practical concerns, not evidence that one ordering always wins.

For the example above, you can settle which project names belong in the result without specifying every class. Start with the observable behaviour, implement a small part, and refine the design as the work reveals more information. If a test depends on private helper names rather than the promised result, a harmless refactor may break it.

Writing tests after implementation can still be useful. Review those tests against the requirement and check that they would expose a relevant defect. See how to check whether green tests prove the product promise when the requirement itself is uncertain.

If your team needs help making these decisions part of everyday delivery, request an AI Delivery Programme conversation.

Frequently Asked Questions