We turned on AI review for every pull request and within two weeks people had stopped reading it. Not because it was wrong — a fair share of the comments were technically correct. Because a 200-line PR came back with 23 suggestions, most of them about naming preferences and missing JSDoc, and somewhere in there was one comment about an unhandled null that nobody saw.
That is the whole problem with automated code review in one paragraph. The bottleneck was never the volume of feedback. It was attention. And a tool that produces 23 comments where a human would leave 3 does not add review capacity, it consumes it.
We kept it, but only after changing almost everything about how it was configured. Here is what worked.
The Only Metric Worth Tracking
Not comments generated. Not files covered. What fraction of the bot's comments result in a code change?
Ours started somewhere around 10%. That number is a description of a tool being ignored. After the changes below it sits in the 50-60% range, and — the part that actually matters — people read the comments now, because experience has taught them the comments are usually worth reading.
If you take one thing from this: measure that ratio before you tune anything else. It tells you whether you have built a reviewer or a spam bot.
Give It One Job
The default configuration of most of these tools is "review this code," which is far too broad. The model has no idea what your team already handles elsewhere, so it comments on everything it can see — including all the things your linter, formatter and type checker already enforce.
The fix is subtractive. Explicitly tell it what not to raise:
# .ai-review.yml
focus:
- Logic errors, off-by-one, inverted conditions
- Unhandled null/undefined and empty-collection cases
- Missing await, unhandled promise rejections
- Race conditions and non-idempotent writes
- Missing authorization checks on new endpoints
- Secrets, credentials or PII in code or logs
- N+1 queries and unbounded result sets
ignore:
- Formatting, spacing, import order # prettier owns this
- Naming preferences # not worth a round trip
- Missing comments or JSDoc
- Type annotations # tsc owns this
- Test coverage percentages
- Suggestions to "consider extracting" without a concrete bug
rules:
- Comment only when you can name a concrete failing input.
- If confidence is below high, say nothing.
- Maximum 5 comments per pull request. Rank by severity.
That last block did more than everything else combined. A hard cap forces prioritisation. When the tool can only say five things, it says the five that matter, and the naming nitpicks fall off the bottom on their own.
The "name a concrete failing input" rule is the other big one. It converts "this could be more robust" — which is unactionable and mildly annoying — into "if items is empty, line 34 divides by zero." One of those gets fixed.
What It Is Genuinely Good At
Being fair to the technology, there are categories where it consistently outperforms a tired human reviewer at 5pm.
Mechanical correctness across a big diff. A rename that missed one call site. A parameter order swapped in one of six invocations. Humans skim large diffs; the model does not get bored.
Error-path omissions. Reviewers read the happy path because that is what the PR description describes. The model reliably notices the promise with no .catch, the response used without checking status, the empty-array case.
Cross-file consistency. "The other three handlers in this module wrap errors in AppError, this one throws raw." That is exactly the kind of thing that needs someone holding the whole module in their head.
Security patterns with a recognisable shape. Interpolated SQL, a new endpoint with no auth guard, a logged token. Not a substitute for a security review, but a decent net.
What It Is Bad At, and Will Stay Bad At
Whether the change should exist. The most valuable review comment I ever received was "this whole feature is solving a problem we decided not to solve." No amount of diff context produces that.
Architectural fit. The model sees the diff, not the two years of decisions that made the codebase this shape. It will confidently suggest a pattern that is technically fine and wrong for your system.
Anything requiring product knowledge. Whether the discount logic matches what finance agreed. Whether that field is actually nullable in production data despite the type.
Knowing when a rule should be broken. Every codebase has deliberate exceptions. The model relitigates them on every PR unless you write them down.
Which is why I am firm on this: the bot does not approve pull requests. It comments. A human approves. The moment it can approve, the review stops being a review.
Where to Put It in the Pipeline
Two placements, and the first is underrated.
Before the PR is opened — the author runs the review locally on their own diff. Fixes happen in the author's head while the code is still fresh, with no public back-and-forth and no round-trip latency. This is where most of the value is, and it is where people are most receptive, because nobody is watching.
On the pull request — as a comment, never a blocking check. A blocking AI review means one confident false positive stops a release, and the first time that happens someone adds a bypass label that then never comes off.
Keep the cost visible too. Reviewing every commit on every branch adds up quickly. Diff-only, on PR open and on subsequent pushes to that PR, is a sane default.
The Thing That Actually Changed Our Review Quality
Slightly against the theme of this post: the biggest improvement was not the AI. It was that the AI made the cost of a large diff obvious.
Once the tool was capped at five comments, everyone could see that a 900-line PR was getting five comments the same as a 90-line one — and that the 900-line PR was clearly not being reviewed properly by anyone, human or otherwise. That visibility did more for our review culture than the suggestions did.
Small PRs, a human who approves, and a machine that catches the null you missed at 5pm. That combination works. Twenty-three comments about naming does not, and it will quietly train your team to ignore the one comment that mattered.
