LLMs are non-deterministic and their context gets polluted over a long session. The orchestration around them should not inherit those properties. Anywhere a step in your workflow can be made exact and machine-checked, make it exact, so the fuzzy part is contained to the model's actual output and can't quietly skip or corrupt the process around it.
PR review is a good place to apply this. An agent running review passes can skip one, forget one of several, or merge over a finding it decided wasn't worth fixing, and a chat-based "all reviews passed" gives you no way to know. This is how to close that gap.
Every PR commits the review results as files a CI check reads. The merge is blocked unless the results are present, complete, current, and clear of unresolved serious findings. The reviews stay probabilistic. Whether they happened and cleared the bar becomes a yes/no that CI answers, and an agent cannot talk its way past a required status check the way it can through a chat turn.
One JSON file per review pass under reviews/pr-<number>/, e.g. code-review.json, security-review.json:
{
"leg": "code-review",
"model": "gpt-5.5",
"reviewed_diff_sha256": "6c314be2ac439d1d1202b713cd27f3601761262d6ff2f18a91b85024facc12bf",
"reviewed_at": "2026-07-21",
"raw_output": "full text of the review pass, verbatim",
"findings": [
{
"id": "F1",
"summary": "retry loop re-reads the response body after it was consumed, so the second attempt sends an empty payload",
"severity_claimed": "P1",
"validated": true,
"severity_validated": "P1",
"disposition": "fixed",
"reason": "buffer the body once before the loop"
}
]
}A CI job, wired in as a required status check, fails the PR unless:
- every expected pass is present and parses
- every finding has a validation verdict and a disposition (fixed, or dismissed with a reason)
- nothing both validated and above a severity line you set (e.g. security high/critical, or a P1 bug) is left unfixed
- the embedded hash matches the current diff
The third rule is the hard stop, and it's why the artifact separates claimed severity from validated severity. A reviewer asserts a severity; you or a dedicated validation pass confirm whether the finding is real and what its severity actually is, and record that verdict with a reason. Only a validated finding at or above your line blocks the merge, so a false-positive "critical" gets dismissed with a reason and doesn't wedge the PR, while a real one can't be relabeled away in prose. Clearing one means fixing it, and the fix restales the reviews through the hash, so you re-review and land a clean pass. The line sits at a serious tier for a reason: reviewers will keep turning up P3/low/info findings without end, and gating on those would never converge, since each fix invites a fresh round of minor nits. Those still get recorded and dispositioned, but they don't block. The loop terminates because it only waits on crit/high/P1 having no open disposition.
Keep the script dependency-free so it runs identically locally and in CI. Validate types before testing membership, or a hand-edited "severity": [] crashes with a stack trace when you wanted a clear error.
The last gate rule is what makes this more than a checkbox. A review of commit A must not vouch for commit B. Bind each artifact to a hash of the diff it reviewed:
git diff <base>...HEAD -- . ':(exclude)reviews'
then sha256 the raw bytes. Excluding reviews/ from the diff is the trick: if the hash covered the review files, committing them would change the hash they attest and you could never reach green; with it excluded, committing artifacts leaves the hash alone while any code edit changes it and marks every artifact stale. That gives you re-review-after-fix without asking: a fix commit fails the gate until the passes run again against the fixed code.
Pin the diff bytes (--no-ext-diff --full-index --no-color, fixed diff.algorithm) so a local git config can't produce a different hash than CI, and enable branch protection's "require branches up to date" so a base advance also restales open PRs.
Run the base branch's copy of the gate script against the PR checkout, not the PR's own copy, so a PR can't weaken the gate by editing the script in the same change.
Merging the PR that adds the gate job does not enforce anything. The job runs and reports, but nothing blocks a merge until you mark it required, which is a repo settings change, not something a PR can carry. So the PR that introduces the gate is itself unprotected by it; enforcement starts on the next PR.
In the repo: Settings, Rules, Rulesets, targeting your integration and release branches. Enable "Require a pull request before merging" and "Require status checks to pass," then add the gate job by its exact check name alongside your existing required checks. Enable "Require branches to be up to date before merging" so a base advance restales open PRs. Leave a repo-admin bypass if you want a human escape hatch; the point is that the agent has no bypass. Or with the CLI:
gh api -X POST repos/OWNER/REPO/branches/BRANCH/protection/required_status_checks/contexts \
--input <(echo '["review-gate"]')
Verify it took, since a wrong check name silently never triggers:
gh api repos/OWNER/REPO/rules/branches/BRANCH \
--jq '.[] | select(.type=="required_status_checks")
| .parameters.required_status_checks[].context'
On other forges the mechanism has a different name (GitLab merge request approvals plus a pipeline gate, Bitbucket merge checks) but the shape is the same: a required, non-bypassable status the merge waits on.
The gate proves the reviews happened, are current, and cleared the bar. It does not prove they are honest: an agent can write a clean artifact for a review that never ran, or mark a real finding fixed when it isn't. This closes the two failures you hit in practice, forgetting a pass and silently dropping a finding, and does nothing against deliberate fabrication. Embedding raw output and keeping the files as an audit trail helps a little. If your threat model includes a malicious committer, this is not that control. Same for the gate's own files: on most CI the workflow comes from the PR branch and a skipped required job reports success, so route changes to the gate script and CI config through human review.
The sequence whoever opens the PR follows, agent or person, once the code is ready. Order matters because any code change after a review invalidates it.
- Land the last code change on the PR. All fixes are in, nothing more to edit.
- Run the review passes against that final state and record their results, including how you validated and dispositioned each finding.
- Compute the hash and write it into each artifact.
- Commit the artifacts. The hash is unaffected, since the
reviews/directory is excluded from it. - Run the gate script locally to confirm it passes, then push. CI runs the same check.
If a reviewer turns up something you fix, that fix is a new code change, so go back to step 2 for the affected passes. The hash will refuse to match until you do.