← Back to the CV

Personal project · agentic code reviewer

A code reviewer that reads every pull request

Every pull request in my homelab is read by a self-hosted LLM reviewer before it merges, including the ones my own coding agents open. Between August and October 2026 it reviewed 786 PRs across 13 repositories in 1,456 passes. This page shows how a pass runs, what keeps it contained, and what keeps it cheap.

How a pass runs

Contained by construction

Kept cheap on purpose

Honest about what it read

When a lens runs out of budget or triage times out, the reviewer posts nothing rather than an all-clear it can't back. Every review states its own scope, such as how much of the diff an incremental pass actually read, so a green check never claims more than it covered.

Measured against its own misses

Changes to the reviewer are re-scored against two sets kept apart. A golden set is built from my own repositories: bugs that slipped past review and were fixed within a week, and cases where the reviewer made a wrong claim someone acted on. A public benchmark of 50 labelled pull requests from large open-source projects is scored separately. Replays run the reviewer against a local fake forge, so nothing posts to a real PR, and a judge from a different model family votes three times on each match.

Recall on the golden set is still low, and that is the point of keeping it: each change has to move the number, not just read better.

Built with

Part of the same homelab as the CV's personal projects; the agents it reviews are on their own page.