4 min read
AI coding productivity has become a contest between impressive numbers.
One study reports faster completion. Another reports a slowdown. A vendor shows accepted suggestions. A team dashboard
shows more merged pull requests. Everyone appears to measure the same thing.
They are not.
The important question is not which number wins. It is which outcome the number represents.
Why credible studies disagree
A Microsoft Research working paper combined three field experiments across 4,867 developers and estimated a 26.08%
increase in completed tasks for developers with access to an AI coding assistant.
A separate Microsoft experiment with 95 developers found participants with GitHub Copilot completed a bounded HTTP
server task 55.8% faster.
METR’s early-2025 study found a different result: experienced open-source developers working in familiar repositories
took 19% longer with AI. In February 2026, METR explained that newer tools probably produce more speedup, but selection
effects and unreliable time reporting made its newer estimate too weak to trust.
These findings do not need to be forced into one universal number.
They involve different developers, tasks, repositories, tools, periods, incentives, and definitions of completion.
The correct lesson is that “AI makes developers X% faster” is usually too broad to guide a team.
Build a measurement stack
Start by separating five layers.
1. Agent activity
Prompts, tokens, agent runs, generated lines, branches, and test executions show how much the system did.
These measures help with cost and capacity. They say little about value.
2. Task throughput
Completed tickets, opened pull requests, and merged changes move closer to delivery.
They can still be misleading. When work becomes cheaper, teams may create tasks they would not have considered worth
doing before. More completed tasks may reflect lower production cost rather than higher product value.
3. Accepted engineering outcomes
Track whether a change was accepted without avoidable correction, how much review time it consumed, and whether its
evidence covered the approved behaviour.
This exposes the work that first-draft metrics hide.
4. System health
Measure escaped defects, rollback rate, structural drift, dependency risk, flaky tests, documentation freshness, and
the team’s ability to explain the changed area.
A short task can create a long maintenance cost.
5. Product value
Connect the change to an intended product result: fewer failed checkouts, faster support resolution, lower operational
risk, improved conversion, or another owned outcome.
Not every engineering task has an immediate customer metric. The team should still be able to explain why the work was
worth accepting.
Use paired measures
Single metrics are easy to optimise and easy to game.
Pair speed with quality:
| Speed measure | Quality or value partner |
|---|---|
| Time to first draft | Time to accepted change |
| Pull requests opened | Pull requests accepted without rework |
| Tests generated | Acceptance criteria actually proven |
| Agent runs | Useful outcomes per run |
| Changes merged | Escaped defects and rollback |
| Documentation written | Documentation verified against live behaviour |
The pair keeps a faster activity from hiding a downstream bill.
Measure by task and risk
AI assistance is unlikely to have one effect across a codebase.
Repetitive, familiar work can behave differently from architecture changes. A small visual adjustment can behave
differently from an authorisation rule. A developer new to a framework can gain speed and lose understanding at the
same time.
Segment the data:
- task type;
- repository familiarity;
- risk lane;
- developer experience;
- model and tool;
- autonomous versus interactive use;
- accepted versus reverted outcome.
Then look for conditions under which the workflow helps.
Keep the measurement honest
Three rules prevent a productivity programme from turning into theatre.
First, record the baseline before changing the workflow.
Second, keep limitations beside every number. A field experiment, benchmark, self-report, and internal dashboard answer
different questions.
Third, ask the people doing review and maintenance what moved. A faster implementation may have transferred work to a
less visible role.
Where orchestration matters
paqad-ai can provide the structure needed for better measurement because its workflow separates stages and records
artifacts: routing, plan, specification, development, review, checks, documentation, human decisions, and evidence.
That makes it possible to measure more than generation.
The tool does not guarantee a productivity result. No honest tool can. It gives a team a more complete operating record
from which accepted outcomes, correction work, evidence gaps, and decision delays can be studied.
Measure activity because it controls cost.
Measure accepted engineering outcomes because they control risk.
Measure product value because that is why the work exists.
Install paqad-ai: https://github.com/Eliyce/paqad-ai/
Source notes
- Microsoft Research, three field experiments,
working-paper estimate of 26.08% more completed tasks across 4,867 developers. - Microsoft Research, GitHub Copilot experiment,
bounded implementation-speed experiment with about 95 developers. - METR, 2026 uplift experiment update, explains why the early
slowdown result should not be generalised to current tools. - METR, Task Substitution and Uplift, separates
task speed from delivered value.
