AI code review is worth keeping, but its false positives are the wrong thing to judge it by.
On my team, about 120 bot comments across roughly 50 pull requests split three ways: about two in five led someone to change the code, about one in five was plainly wrong, and the rest were true but not worth a change. That third pile is what got the bot ignored.
The two kinds of noise answer to different fixes. Wrong comments are about code the reviewer can’t see, so only showing it the callers and your team’s rules removes them. The true-but-pointless pile comes from a reviewer that says everything it notices.
I tried each fix alone, running Claude Code three times on six planted-bug diffs. A prompt that named risk areas and said to skip style still left 21 of its 37 comments in that pile, averaged over the runs. One more sentence telling it to stay quiet unless it’s sure cut the pile to 7.
The bot still caught all five planted bugs every time, but it also went quiet about missing tests.

On this page
What 120 AI review comments from 50 PRs turned into
Bot threads resolved without a reply
I saw the problem before I counted anything. In the PR conversation tab, the bot’s threads had no replies, only a click on Resolve conversation, and on one PR a handful had been closed seconds apart.
Then a teammate said it out loud in a review: half the bot’s comments are about variable names, so they resolve them all and read the human ones. What got closed with the names was the real stuff: a possible null, a stream left open, a new branch with no test.
One thread flagged a service method that caught an exception from an external call, logged it and returned an empty list. I found that resolved thread only after the code reached production, where the logs showed it quietly serving empty screens.
The slice that was right and not worth a change
So I sorted those comments, from a GitHub Copilot-based reviewer and our in-house LLM reviewer, by one question: did a person change the code because of it? I expected the wrong ones to be the problem. The bigger slice was comments like “this variable name is ambiguous; consider a name that shows its role.”
That one wasn’t wrong, but the name matched what the team called that thing in the nearby modules. Renaming it in one place breaks the match, and renaming it everywhere drags unrelated files into the diff and hides the logic change.
Restated house style and null checks on values that can’t be empty sat in the same slice. A comment like that isn’t a false positive; it’s a cost. Each one is harmless, but together they make every bot comment look alike, and the null check gets resolved along with the rename.
That’s also why the two in five that led to a change aren’t an accuracy score: the rest weren’t all wrong. Our rule now is that a bot thread gets a fix or a one-line reason before anyone resolves it.
Six planted-bug diffs: callers removed the wrong comments, one line cut the nitpicks
To see which change moves which slice, I ran Claude Code headless (claude -p) against six small Python diffs, five with one planted bug each, three times per prompt. Only the lines after the review request changed between prompts.
Wrong comments: the diff hides the caller’s check
With the diff alone, every wrong comment was about input the caller had already checked. In the paging diff, the bot found the planted off-by-one first, then asked for validation of a page value the route already range-checks.

The bot wasn’t misreading the code. It reasoned from what it could see, and the diff doesn’t show the check. I’d seen the same thing on my team, when a reviewer looking only at a controller change flagged a missing authorization check that our API gateway and a Spring Security filter already enforced.
Pasting the caller file into the prompt along with three team rules removed all of them in every run:
Team rules: 1. Short local names (u, n, s, uid, out) are fine in functions under 30 lines. 2. Request input is validated in the API layer (see callers); service functions do not re-validate it. 3. A broad `except Exception` is allowed when the line carries an `# intentional:` comment.
Swap in your own team’s allowances and limits; the second rule is the one that names where the check lives.
Where that context goes depends on the reviewer:
- With
claude -por an in-house prompt, paste the caller and the rules in, as I did. - GitHub Copilot code review can already read the rest of the repository during a review, so what it lacks is the rule saying which layer validates; that goes in
.github/copilot-instructions.md, or in a.github/instructions/*.instructions.mdfile scoped to the service code. - For a vendor bot, look for its repository instructions file and put the same lines there. If the bot reads only the diff, those lines are the only context it gets, so make the second one name the layer that does the checking. If it takes no instructions at all, it can get neither the callers nor your rules, so move the review to one that does.
A risk-area list renamed the nitpicks
This scope line shortened the list but barely changed its mix, since more than half of what it left was still true and not worth a change:
Only comment on security, concurrency, error handling, data integrity, or performance/resource usage. Skip style and naming.
In every run some nitpicks took the allowed names. In the first, a numeric column got a “CSV formula injection” warning and a name collision became an “impersonation risk”; later runs turned an uncapped list of user IDs into “resource exhaustion”. It still caught the off-by-one every time, though it wasn’t on its list, and in two of the three runs called it a data-integrity issue.
Adding a stay-quiet sentence
Adding one sentence after that line is what shrank the pile:
If you are not confident an issue is real, do not mention it.
The relabelled nitpicks went with it. The same two wrong comments survived in every run, both about validated input, so the reviewer was confident about code it couldn’t see. But the line also dropped the “this path has no test” comments, and those were real.
Without the stay-quiet line, the bug-free diff still drew 3 to 11 comments a run, as many as the buggy ones with the diff alone, so a long list says nothing about whether a PR is broken. Six diffs and three runs per prompt show which slice each change moves, not Claude’s accuracy.
What to change, in order
- Give the reviewer the callers, or whatever layer validates its input, and your team’s allowed-style rules.
- Then put the scope line and the stay-quiet line, in that order, at the end of your review prompt, since that pair is what I tested. On my six diffs this pair took 54 comments down to 19 on average, so expect clearly fewer bot threads per PR, with the “no test” comments among the ones that go.
- Check for missing tests somewhere that doesn’t depend on the bot’s confidence. My team moved them to a changed-line coverage check in CI.
On a Python project, diff-cover reads the XML from pytest --cov --cov-report=xml and runs as diff-cover coverage.xml --compare-branch=origin/main --fail-under=80. Set the 80 to the share of changed lines your team expects new code to cover.
On another stack, check whether your coverage tool can write a Cobertura, Clover, JaCoCo or LCov report, since those are the formats diff-cover reads.
A four-area scope prompt, the N+1 it missed, then five reviewers
What one scope paragraph cut
My team’s first narrowing was one paragraph in our in-house bot’s prompt: four risk areas (security, concurrency, error handling and data integrity), no comments on naming, style or minor refactors, and if you aren’t sure, say nothing. That last clause is the same stay-quiet line.
Comments about names and over-defensive checks thinned out, and people started replying to bot threads again.
It didn’t stop the comments it couldn’t judge. Some still said things like “no authorization check” on a method behind a gateway and an interceptor that already handled it. Because those comments said security, people wrote replies explaining why, and that annoyed them more than the names had.
They stopped only later, when security got its own reviewer whose prompt said checks may happen upstream and such findings should be marked low-confidence. That was context again, not wording.

The N+1 the silence let through
Narrowing also had a price. Weeks later, a list endpoint shipped with a lookup inside a for loop, a plain N+1, and the tests, run on a handful of rows, passed.
Performance was out of scope, so the bot said nothing, and the approver read that silence as all clear. Staging caught it when the endpoint slowed to over a second.
Five reviewers and a note on what was checked
After that, performance and resource use became a fifth area, and each area got its own reviewer, because one reviewer covering four had tangled lock scope and transaction boundaries into a single comment. A final step merges overlaps, drops anything both low-severity and low-confidence, and sorts the rest by severity.
The bot also keeps one pinned comment on every PR: which five areas it checked, that it didn’t judge design or test coverage, and that zero findings means nothing turned up in those five, not that nothing is wrong.
Counting your own team’s split before comparing it to Uber’s 65%
Take 20 to 30 recent PRs and ask of each bot comment whether a person changed the code because of it. Three things keep that count from inflating:
- Don’t trust the resolved state, since GitHub lets the PR author or anyone with write access resolve a conversation. Open the commits after the comment and check whether the flagged line changed in a way that deals with what it said.
- GitHub marks a conversation outdated when its line changes for any reason, formatting included, so use that only to decide where to look.
- Count a follow-up PR that links back as a fix, and a different fix for the same problem too. When you can’t tell, file it as low value, or your acted-on share inflates.
Calling a comment wrong takes longer, because you have to find the layer that already handles it. A PR took me a few minutes, unless it held a wrong comment to chase down.
Only then put your number next to the published ones, after reading what each one counts:
| Number | What it counts | Who decides |
|---|---|---|
| About 40% (my team) | A person changed the code because of the comment | Me, reading the commits after each comment |
| 65% addressed (Uber uReview) | The comment doesn’t come back when the reviewer reruns five times on the final commit | The reviewer itself |
| 73.8% resolved (a 2024 study at the appliance maker Beko) | Comments labelled “Resolved” | The PR author |
The same pile of comments can honestly score 40% by my rule and 65 to 74% by a rerun or a label. Don’t measure it against 100% either: at Uber, human reviewers’ comments were addressed about half the time.
Compare against your own number, recounted the same way next month after the reviewer has the callers and the stay-quiet line.
Sources
My measurement:
– Research note on loopnotes-lab: the six diffs, the prompts, every comment with its label, and the classification rules (Claude Code 2.1.327, model claude-opus-5-5, three runs per prompt, November 18–19, 2026)
Official pages:
– Commenting on a pull request, GitHub Docs, checked 2026-11-18: Resolve conversation and who can resolve it
– Precision review comment outdating, GitHub Changelog, 2018-09-24: a conversation turns outdated when its line changes
– CLI reference, Claude Code Docs, checked 2026-11-18: claude -p headless mode
– About GitHub Copilot code review, GitHub Docs, checked 2026-11-18: repository context during a review, .github/copilot-instructions.md and path-specific .instructions.md files
– diff-cover README, checked 2026-11-18: --compare-branch, --fail-under, Cobertura and JaCoCo XML input
Other teams’ reports:
– uReview: Scalable, trustworthy GenAI for code review at Uber, Uber Engineering, 2025-08-12: 65% addressed (five reruns on the final commit), 51% of human comments addressed
– Automated Code Review In Practice (arXiv 2412.18531), 2024-12-28: Beko’s reviewer built on Qodo PR-Agent, 73.8% “Resolved” and 21.3% “Won’t Fix” as labels the PR author set