An automated reviewer that reads every pull request, compares what changed against what the ticket actually asked for, and leaves one comment saying where those two things disagree. It also flags the security problems that are easy to miss in a diff. It never blocks a merge — a person still decides.
Scope and ownership. Platform team of six engineers and a manager, supporting four product teams. I did not build the underlying action — that's Anthropic's, off the shelf. What I built is the review itself: what it looks for, what it deliberately ignores, how it gets the ticket, and the shape of the output. The distinction matters, and I'd rather state it than have it found.
Every pull request now gets a first-pass review before a human opens it.
Method: coverage is the honest claim here. Every PR is reviewed because the trigger fires on every PR — that's a property of the wiring, not a measurement. I never instrumented reviewer time saved, so I don't quote a number for it.
The motivating problem was that AI-assisted development had made writing code much faster without making reviewing it any faster. Review became the bottleneck. This takes the mechanical part of the first pass — did the diff do what the ticket said, are there obvious risks — so a human starts from a shorter list.
The gap I'd close first: I never measured a false-positive rate. "Engineers quietly stopped reading it" is the failure mode that kills tools like this, and I have no instrument that would have told me it was happening. Sampling fifty reviews and labelling them by hand is what I'd do before expanding it further.
The workflow runs when a pull request is opened, resolves the ticket key from the PR description, fetches that ticket, reads the diff, and posts a single comment in three fixed sections: line-level findings, a requirements checklist, and a verdict.
Three decisions shaped it, and each one traded something away.
Advisory, not a required check. A non-deterministic reviewer should not hold the merge button. Two reasons: the fastest way to make engineers resent a tool is to have it block them on a false positive, and a required check implies a guarantee a model cannot make. The deterministic checks — tests, linters, scanners — stay blocking. This one is a second pair of eyes.
It runs when a PR opens, not on every push. Re-running on every commit multiplies spend with no proportional benefit. The cost I accepted is that the review goes stale after the first round of fixes, so it never sees problems introduced by responding to its own feedback. If spend weren't a constraint I'd run it on ready-for-review plus explicit re-request.
The prompt lives inline in the workflow rather than being read from a file at runtime, so changing the review contract requires a merge to the default branch like any other protected change. Single-sourcing the content across repos through a generator keeps the copies identical without weakening that boundary.
All three decisions live in a handful of lines of the workflow:
on:
pull_request:
# opened/reopened only — NOT synchronize. Cost control.
types: [opened, reopened]
permissions:
contents: read
pull-requests: write # comment only. No merge, no push.
steps:
- name: AI review
uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: | # inline, so changing it needs a merge
...But the part that took the most iteration wasn't what to catch — it was what to stay quiet about:
The first version commented on everything it noticed, and that nearly killed it.
A reviewer that flags a naming inconsistency in the same voice it flags a wildcard IAM policy teaches people that most of what it says is skippable — and once someone is scrolling past its comments, they scroll past the one that mattered too. The tool wasn't wrong. It was noisy, which in review tooling is the same thing as being wrong, just slower-acting.
The fix was to write the silence down. An explicit IGNORE list names what the reviewer must never comment on, on the grounds that linters and formatters already own those and do it deterministically. The AWS account ID entry is the one I'd point at: account IDs look like secrets, and a security-minded reviewer flags them every time — but IRSA role annotations always contain one, so flagging it means crying wolf on the standard pattern in nearly every infrastructure PR.
I'd make the same call again, because you cannot calibrate this before seeing real output — the first version has to be over-eager so you can find out which of its instincts are wrong. What I'd change is treating the IGNORE list as a first-class part of the review contract from the start, rather than something bolted on once engineers started ignoring the bot.