An automated reviewer that reads every pull request, compares what changed against what the ticket actually asked for, and leaves one comment saying where those two things disagree. It also flags the security problems that are easy to miss in a diff. It never blocks a merge — a person still decides.
Platform team of six engineers and a manager, supporting four product teams. I did not build the underlying action — that's Anthropic's, off the shelf. What I built is the review itself: what it looks for, what it deliberately ignores, how it gets the ticket, and the shape of the output. The distinction matters, and I'd rather state it than have it found.
Every pull request now gets a first-pass review before a human opens it.
coverage is the honest claim here. Every PR is reviewed because the trigger fires on every PR — that's a property of the wiring, not a measurement. I never instrumented reviewer time saved, so I don't quote a number for it.
The motivating problem was that AI-assisted development had made writing code much faster without making reviewing it any faster. Review became the bottleneck. This takes the mechanical part of the first pass — did the diff do what the ticket said, are there obvious risks — so a human starts from a shorter list.
The gap I'd close first: I never measured a false-positive rate. "Engineers quietly stopped reading it" is the failure mode that kills tools like this, and I have no instrument that would have told me it was happening. Sampling fifty reviews and labelling them by hand is what I'd do before expanding it further.
The workflow runs when a pull request is opened, resolves the ticket key from the PR description, fetches that ticket, reads the diff, and posts a single comment in three fixed sections: line-level findings, a requirements checklist, and a verdict.
Three decisions shaped it, and each one traded something away.
Advisory, not a required check. A non-deterministic reviewer shouldn't hold the merge button: blocking on a false positive is the fastest way to make engineers resent a tool, and a required check implies a guarantee a model cannot make. Tests, linters and scanners stay blocking. This is a second pair of eyes.
It runs when a PR opens, not on every push. Re-running on every commit multiplies spend with no proportional benefit. The cost I accepted: the review goes stale after the first round of fixes, so it never sees problems introduced by responding to its own feedback.
The prompt lives inline in the workflow rather than being read from a file at runtime, so changing the review contract requires a merge to the default branch like any other protected change. Single-sourcing the content across repos through a generator keeps the copies identical without weakening that boundary.
All three decisions live in a handful of lines of the workflow:
on:
pull_request:
# opened/reopened only — NOT synchronize. Cost control.
types: [opened, reopened]
permissions:
contents: read
pull-requests: write # comment only. No merge, no push.
steps:
- name: AI review
uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: | # inline, so changing it needs a merge
...But the part that took the most iteration wasn't what to catch — it was what to stay quiet about:
The first version commented on everything it noticed, and that nearly killed it.
A reviewer that flags a naming nit in the same voice it flags a wildcard IAM policy teaches people its comments are skippable — and then they scroll past the one that mattered. It wasn't wrong. It was noisy, which in review tooling is the same thing, just slower.
The fix was writing the silence down. An explicit IGNORE list names what the reviewer must never comment on, because linters already own those and do it deterministically. The entry I'd point at is AWS account IDs: they look like secrets, but IRSA role annotations always contain one — so flagging them means crying wolf on nearly every infrastructure PR.
I'd make the same call again — you can't calibrate this before seeing real output, so the first version has to be over-eager. What I'd change is treating the IGNORE list as part of the review contract from the start, not something bolted on once engineers began ignoring the bot.