4 min read
Two engineering leaders can cite honest studies and reach opposite conclusions about AI coding productivity. One sees faster bounded tasks. The other sees experienced maintainers slowed inside familiar repositories. The contradiction disappears when you inspect the unit of work, repository context, developer experience, and outcome measured. This is an explicitly fictional composite, used to expose a repeatable engineering decision without inventing a customer result.
AI coding productivity is conditional, not one number. Field experiments report more completed tasks, a bounded Copilot experiment reported faster completion, and METR observed experienced maintainers taking longer on familiar-repository tasks. Measure generation, verification, delivery stability, and maintenance separately before deciding whether your own workflow improved.
Different studies answer different questions
The 4,867-developer field experiments reported a 26.08% increase in completed tasks with assistant access. A separate controlled experiment with 95 developers reported 55.8% faster completion on one JavaScript task. Those results establish that gains are possible. They do not establish the same gain for a mature codebase, sensitive feature, or long maintenance horizon.
Field experiments with 4,867 developers helps bound this point. The working paper measures completed tasks, not long-term maintainability or security.
| Lens | What to inspect |
|---|---|
| 1 | Field: 4,867 developers |
| 2 | Bounded task: 95 developers |
| 3 | Familiar repos: 16 maintainers |
| 4 | Repository pilot: 4 weeks |
The numbers and labels above are a diagnostic, not benchmark data. Replace them with repository evidence before using the model in a staffing or investment decision.
A four-column scorecard keeps the claim honest
Compare evidence using four columns: participants, task setting, measured outcome, and limitation. Add the date because model and tool behavior changes. This prevents a leadership slide from placing a short greenfield task beside a quarter of delivery data as if they measured the same thing.
The sequence matters because later evidence depends on earlier intent. Skipping one step transfers uncertainty to a reviewer who has less time and often less context.
Perceived speed can diverge from measured time
METR studied 16 experienced maintainers across 246 tasks in repositories they knew and reported a 19% slowdown, despite developers expecting and perceiving gains. The authors later noted that newer evidence is uncertain and may differ. The useful lesson is measurement discipline, not a slogan that AI always helps or always hurts.
METR experienced developer study adds a second boundary. The study covered 246 tasks in familiar repositories and should not be generalized to every team or current model.
:::mistake **Mistake:** Using one study headline as a forecast for every repository. **Fix:** Match each result to its participants, task, tool date, and measured outcome. :::The correction is deliberately procedural. A workflow can be inspected, rehearsed, and improved. A warning without an owner or artifact rarely survives the next busy sprint.
Run a repository-level pilot before rollout
Run a four-week pilot in one repository. Capture time to first draft, reviewer minutes, time to merge, reopened work, escaped defects, and documentation completion. Compare similar work classes and keep the model version fixed during the measurement window. Then decide which tasks earned continued use.
paqad-ai v1.67.0 was the current public release when this article was verified on July 21, 2026. Its public repository describes local workflows, risk routing, specialist roles, deterministic checks, documentation sync, and audit records. Those are product mechanisms, not independent proof that a team will achieve a specific outcome.
Use the broader AI workflow audit guide to map the operating system, compare the evidence bar with production-ready AI code, and use the consultant selection guide when outside ownership is being considered.
The decision rule for this article is: Adopt an AI coding workflow only for task classes where repository-level evidence shows a net gain without a worse risk signal.
Frequently Asked Questions
Why do AI coding studies disagree?
They study different developers, tasks, repositories, tools, and outcomes. A short bounded task measures something different from maintenance inside a familiar repository. Read the method before carrying the headline into a staffing or delivery forecast.
What metric matters most?
Use a small set together: cycle time, qualified reviewer time, reopened work, escaped defects, and documentation completion. A single metric can move work downstream and still look positive.
How long should a pilot run?
Four weeks is enough to expose obvious queue and review effects for an active team. Longer horizons are useful for maintenance, but keep the first pilot narrow enough that tool, model, and process changes do not blur the comparison.
What next?
If this failure pattern exists in your repository, install paqad-ai and test the decision tool above on one real feature. Keep the evidence local, inspect provider permissions, and retain human authority for the final risk decision.
