How our reviewer caught two real regressions, blind — a transparent case study
Why we ran this test
Every AI code review tool claims it catches bugs. Almost none of them show you the receipts on code they had never seen, judged against what actually happened in production.
So we set up a test we couldn't fudge.
We went to excalidraw, a large open-source project, and looked for pull requests with a specific property: they were reviewed by a skilled human, approved, merged — and later reverted because they introduced a regression. A revert-after-merge is about as close to ground truth as you get in review. It means the problem was real enough to undo, and that a competent human reviewer looked straight at the diff and missed it.
Then we ran our reviewer on the original diff, blind — no knowledge of the outcome, no hindsight, no hint that anything was wrong. Each review took roughly 2.5 to 4 minutes.
The point isn't to dunk on excalidraw's maintainers. Their review process is better than most. The point is that hindsight makes every bug look obvious, and the only honest way to test a reviewer is to remove the hindsight and see what it does with the same information a human had at the time.
This connects to how we think the whole category should be judged — on real behavior on real code, not demos. We wrote more about that in how to evaluate AI QA tools.
The two jobs
A reviewer has two jobs, and most tools only do the first one badly:
- Catch what matters — the regression that a human missed.
- Stay quiet on correct code — no invented defects, no noise.
The second job is the one that gets tools muted. A reviewer that flags everything eventually flags nothing, because the team stops reading it. We cover that failure mode in why teams mute AI code review.
This test has three cases: two that exercise job one, and one that exercises job two.
Case 1 — Refactored image-cache invalidation
The diff was a refactor of how the image cache decides what to invalidate. Clean, reasonable-looking, the kind of change that reads fine on a first pass.
Our reviewer's top finding: the refactor had silently broadened the cache filter from "new files" to "all files." Instead of re-decoding only images that were just added, the new code re-decoded every image on each add. It didn't just flag the smell — it identified the exact broadened condition and proposed the specific fix to narrow it back.
Ground truth: this PR was merged, then reverted for "flickering and slow performance" — exactly the symptom you'd expect from re-decoding every image on every add.
Result: CAUGHT.
Case 2 — "Optimize" a drag handler
This one was labeled as an optimization to a drag handler. The intent was to make dragging smoother.
Our reviewer's major finding was that the change did the opposite: it forced synchronous re-renders on every drag frame. That's a performance regression dressed as a performance fix — the kind of thing that's easy to approve because the stated goal and the actual effect point in opposite directions. The reviewer also surfaced a second, separate issue in the same diff.
Ground truth: merged, then reverted to undo exactly that change.
Result: CAUGHT.
Case 3 — A 2-line resize-math fix
Here's the case that matters most, and it's the boring one.
The diff was two lines of resize geometry. Our reviewer didn't take our word that something must be wrong. It derived the geometry independently, worked through the math, and concluded the change was correct. It declined to invent a defect. The one thing it did flag was honest and real: no test covered the changed path.
Ground truth: the code was correct.
Result: STAYED QUIET — correctly.
This is the half most tools fail. Point a noisy reviewer at correct code and it will manufacture a concern to justify its existence. Ours looked at a correct change, said so, and limited itself to the one true gap. That restraint is not a smaller achievement than the two catches. It's the reason a team would leave the reviewer turned on.
What this proves — and what it doesn't
Let's be precise, because this is a proof page and precision is the whole point.
- It's n=3. Three pull requests, one project. This is a capability demonstration, not a benchmark.
- It's not a false-positive rate. One correct-code case tells you the reviewer can stay quiet, not how often it does across a codebase.
- Your results will differ. Every codebase has its own patterns, its own conventions, its own definition of "regression." What we measured here does not transfer automatically to your repo.
What it does show is meaningful on its own terms:
- On two diffs that a skilled human reviewed, approved, and merged, our reviewer — working blind — named the regression that later forced a revert, and proposed the fix.
- On a correct diff, it stayed quiet rather than inventing a problem.
- It did both from the same information the human had at the time, with no knowledge of the outcome.
Catch what matters, stay quiet on the rest. Two out of two on the catches, and it held its tongue on the one that didn't need flagging.
See it on your own PRs
The only test that counts is on your code. In a paid pilot we run the reviewer on your real pull requests, in your own environment, with a human approving every action — and we measure the actual false-positive rate on your codebase rather than asking you to trust ours.
Want to see what it catches on your PRs? Start a pilot.