Personal project · agentic code reviewer
A code reviewer that reads every pull request
Every pull request in my homelab is read by a self-hosted LLM reviewer before it merges, including the ones my own coding agents open. Between August and October 2026 it reviewed 786 PRs across 13 repositories in 1,456 passes. This page shows how a pass runs, what keeps it contained, and what keeps it cheap.
How a pass runs
A webhook, not a workflow per repo
One always-on dispatcher receives the forge's signed webhooks and starts a review run for opened, updated and reopened PRs; an hourly sweep catches anything it missed. Reviewed repositories carry no workflow file and no secrets of their own.
Triage picks the lenses
A single call reads the diff and decides which of three lenses it needs:
safetyfor secrets and data loss,bugsfor correctness,infrafor ports, names and producer/consumer drift. Infra paths, secret-like content and very large diffs force their lenses on. The chosen ones run in parallel.A coordinator, then a skeptic
A coordinator merges duplicate findings, checks each against the code and drops nitpicks. A verifier then marks every blocking finding confirmed, refuted (with a file and line) or unsure, within a per-PR token budget.
Findings an author can settle
Findings land as inline comments graded critical or warning, with one summary comment rewritten in place each pass. Each carries a fingerprint, so a fixed finding disappears and a by-design one is dismissed with a reason on the line it concerns, instead of coming back every round.
Contained by construction
The model never runs PR code
Its tools only read: files, directory listings, search and git history. Absolute paths,
..and symlinks that escape the checkout are refused.A PR can't change the rules it is judged by
The reviewer runs from its own repository, never the reviewed one, and per-repository settings are read from the base branch.
PR text is untrusted input
Prompts treat titles, descriptions and diffs as data, and escape anything shaped like a prompt section so a PR can't inject one.
Each run is a fresh container
A pinned image on a dedicated runner, gated on a trusted author and the default branch, failing closed when a check can't be made.
Kept cheap on purpose
Two checks that cost nothing
A pass ends before any model call when the tree is identical to one already reviewed, or when no changed file is a reviewable type.
Only the new part
A follow-up push reviews just the diff since the last reviewed commit, falling back to a full pass after a force-push, a base merge or open findings.
One prefix, many lenses
Every lens gets the same system prompt and diff, with its own instruction last, so the provider's prefix cache serves the diff to the second and third lens. Routing to only the lenses a diff needs cut input tokens by about 95% on the PRs I measured.
Hard caps
Tool calls per lens, a run time budget, two attempts per commit and a per-PR verifier budget, so a large diff costs a known amount rather than an open-ended one.
Honest about what it read
When a lens runs out of budget or triage times out, the reviewer posts nothing rather than an all-clear it can't back. Every review states its own scope, such as how much of the diff an incremental pass actually read, so a green check never claims more than it covered.
Measured against its own misses
Changes to the reviewer are re-scored against two sets kept apart. A golden set is built from my own repositories: bugs that slipped past review and were fixed within a week, and cases where the reviewer made a wrong claim someone acted on. A public benchmark of 50 labelled pull requests from large open-source projects is scored separately. Replays run the reviewer against a local fake forge, so nothing posts to a real PR, and a judge from a different model family votes three times on each match.
Recall on the golden set is still low, and that is the point of keeping it: each change has to move the number, not just read better.
Built with
- Node.js
- Forgejo Actions
- Docker
- DeepSeek
- LLM tool-use loops
- LLM-as-judge evals
Part of the same homelab as the CV's personal projects; the agents it reviews are on their own page.