TL;DR
Teams buying an AI code reviewer hear strong vendor claims, while developers who used one for a year report learning to ignore most of its comments. A leader needs a verdict from their own code. Sort the comments your reviewers write today, replay the tool on past pull requests where the defects are already known, count what it caught and what was noise, then try it on live work with a person still approving. The team then keeps, narrows or drops the tool on evidence.
In this article
- What the published evaluations measured, and why they cannot decide for you
- Sort the comments your reviewers already write
- Replay the tool on pull requests where you already know the answer
- Read the result as three numbers
- Then try it on live work, with a person still approving
- Where the Programme fits, and where it does not
- Frequently Asked Questions
8 min read
Sometimes, on a narrow slice of the work, and how large that slice is can only be measured on your own pull requests. An AI reviewer can take over part of the first pass for likely bugs and agreed conventions; it does not take over design judgment, business context or the decision to merge. Whether it earns even that slice depends on how often its comments are right on your code, which a demo cannot tell you.
That is the question practitioners keep asking. In May 2025 an Ask HN post asked people using AI review on big projects whether it catches bugs or only nitpicks, and whether anyone had stopped asking teammates for review because of it. Its only reply came from a vendor promoting its own product. In August 2026 a developer on the Cursor forum running two AI reviewers on the same pull requests reported getting two separate reviews that covered the same ground, and asked whether a custom reviewer was worth it for everyday code review or whether the managed tool did the job better.
What the published evaluations measured, and why they cannot decide for you
One industrial study followed an LLM review tool built on Qodo PR Agent in the software division of Beko, a home appliance company, across 4,335 pull requests in three projects (Cihan et al., arXiv 2412.18531, December 2024, ICSE 2025 industry track). 73.8% of the automated comments were labelled as resolved, and average pull request closure time rose from 5 hours 52 minutes to 8 hours 20 minutes, with different trends per project. Practitioners also reported faulty reviews and irrelevant comments. Useful feedback and extra waiting arrived together.
A study of open-source pull requests at MSR 2026 compared changes reviewed only by code review agents with changes reviewed only by people (Chowdhury et al., arXiv 2604.03196, April 2026). Agent-only pull requests were merged 45.2% of the time against 68.4% for human-only ones, and 12 of the 13 agents averaged under 60% of comments the authors classed as actionable. The authors state that their correlation does not show the agents caused the abandonment.
Benchmarks add a third kind of number. GitHub published ReviewBench on 5 October 2026: 219 public pull requests, findings scored by an AI judge against a labelled set, results split by severity and by preference for precision or recall (GitHub blog). It is a research preview run by a vendor whose own reviewer is on the leaderboard, and GitHub notes that a fixed answer set grows incomplete as tools find issues its authors missed. Several other review vendors publish benchmarks of their own, each with different pull requests, judges and scoring, so their rankings cannot be compared with each other.
None of these studies used your languages, your conventions or your defect history. They tell you which measures matter: comments acted on, defects caught, noise and waiting time. The values have to come from your repository.
Sort the comments your reviewers already write
Before testing a tool, decide what you would ask it to do. Take the human review comments from your last 30 or so merged pull requests and put each in one of four groups:
- Rule violations that a script could test: naming, forbidden calls, missing migrations, logging format.
- Likely bugs: an unhandled null, an off-by-one, a race, a missing permission check.
- Design questions: is this the right place for the logic, should this be two changes.
- Context the tool cannot know: a customer contract, a planned deprecation, why the code is odd on purpose.
The first group belongs in deterministic checks, validated against passing and failing examples, not in a probabilistic reviewer that may flag a rule on one run and miss it on the next. The case for that is made in executable AI rules. The fourth group stays with people however good the tool becomes. The load an AI reviewer can realistically take is mostly the second group and the easier end of the third. If those are a small share of what your reviewers write, the ceiling on time saved is low before any trial starts.
Replay the tool on pull requests where you already know the answer
A fair trial uses past work, because there you already know what should have been found. Build three sets:
- Historical defects. Pull requests that were merged and later needed a fix: find them through bug tickets, reverts and hotfix commits. Write down the defect each one contained before running anything.
- Seeded defects. Copies of clean pull requests with one realistic defect added by someone outside the trial, mixed across easy and hard cases.
- Clean pull requests. Changes that never caused trouble. A tool that comments heavily on these is showing you its noise.
Run the tool on a fork or a scratch branch with its default settings first, then again with your own instructions: both GitHub Copilot and CodeRabbit read repository instructions, and CodeRabbit offers quieter and more assertive review profiles (CodeRabbit configuration reference). Then have two reviewers label every comment without knowing which set the pull request came from: correct and worth acting on, correct but trivial, or wrong.
Read the result as three numbers
From the labels, count three things per tool and configuration: the share of known defects it caught, split by easy and hard; the share of its comments your labellers would act on; and the number of wrong comments per pull request, with the clean set reported separately.
A trial can mislead in predictable ways. Seeded defects drift towards the obvious kind, so a tool looks better than it will on subtle bugs. A tool that posts many comments catches more by volume while costing more attention, which is why acted-on share and wrong comments per pull request sit beside the catch rate. Labellers who know where the defect is will credit vague comments that happen to point near it. And a result goes stale: GitHub’s documentation says Copilot’s review is not guaranteed to spot every problem and sometimes makes mistakes (GitHub Docs), and the models behind every such tool change, so rerun the replay after major updates.
Decision guide
- Keep it if it catches a meaningful share of the hard defects and reviewers would act on most of its comments.
- Narrow it to the paths or defect types where it did well, and switch it off elsewhere.
- Drop it if the clean set shows more wrong comments than your team will keep reading.
Write the thresholds down before the replay, so the team is not tempted to adjust them to fit whatever it found.
Then try it on live work, with a person still approving
A replay shows what the tool can find; live work shows whether people use it. Run two to four weeks on real pull requests with human approval unchanged, and track how many comments are acted on, what reviewers say about time spent and any defect that escaped despite both.
Check one setting before this phase, not after. Since 1 September 2026, GitHub lets administrators allow Copilot to submit approvals that count toward required reviews, off by default, in public preview and restrictable by file path (GitHub changelog). CodeRabbit’s optional request-changes workflow approves a pull request once its own comments are resolved. Either can change who effectively signs off a merge. Whether that is ever allowed, and for which paths, is the engineering leader’s decision, made in writing rather than inherited from a default.
Cost belongs in the same review. GitHub estimates a Copilot review at about $0.05 to $1 of AI credits at its lighter effort level and $0.25 to $5 at the balanced level, rising with pull request size. Set that against reviewer hours and the value measures you already report, not against comment counts.
Where the Programme fits, and where it does not
The AI Delivery Programme works with one team on one application for five weeks. In week 1 the team records how review works today, including load and defects. In week 2 its testable rules become deterministic checks; the framework’s rule analysis skips what a linter already enforces and classifies the rest as deterministic, heuristic or unverifiable, which narrows what an AI reviewer would be asked to do. Weeks 3 and 4 deliver real backlog work, where a review tool can be trialled, and week 5 compares the outcome with the baseline in writing.
Eliyce does not sell or recommend a review vendor and has no measured comparison of review tools on a customer codebase. If your reviewers are buried and you want the decision made on your own pull requests, explore the Programme through a fit conversation.
Frequently Asked Questions
Can AI code-review tools take real load off human reviewers?
They can take part of the first pass, mainly likely bugs and conventions that are hard to script, if your own trial shows their comments are usually right. They do not remove the human approval, and the Beko study above found pull requests took longer to close after the tool arrived, so measure waiting time as well as comments.
Which review comments are worth handing to a tool at all?
Start from the sorted sample of your reviewers’ comments. Anything a script can test should become a check, because a check gives the same answer every time. Comments that depend on a customer agreement, a planned change or the history of a module should stay with the people who know it, and the tool should not be judged on them.
What does a team give up if reviewers start trusting the tool?
Mostly attention and shared knowledge. Noisy comments train people to skim, the trap the Cursor forum poster met when two tools piled overlapping reviews onto one change, and a confident wrong comment can be accepted unread. Review is also where people learn each other’s code, so keep human review on design changes. Check where the vendor processes your code and what it retains before the trial, not after.
How could a trial on past pull requests mislead us?
Through easy seeded defects, through rewarding comment volume, through labellers who know the answer, and through a small sample of one kind of change. Mix defect difficulty, label blind, include clean pull requests and report results by type of change rather than as one number.
Who decides whether an AI review may ever count as an approval?
The engineering leader who owns the merge rule, after the live trial and with the leads who review most. Record the decision, the paths it covers and the date it will be reviewed again. Without that decision, the default stays: a person approves.
