How to review AI-generated code: by risk, not by diff size

In this lesson you'll learn:

• how to measure a review queue, and why capacity is not the lever;
• how to band every change by what it touches, and enforce the band with a path rule;
• how to open a no-reader lane safely, and the three numbers that keep the policy alive.

Sound familiar?#

• Nine pull requests, a four-day queue, and two senior reviewers who cannot read any faster.
• A one-line change to a payment limit got one quick approval because it was small.
• The review that happens is a scroll to the bottom and an approve.

The policy treats every change as equally dangerous. Agents produce changes faster than people can read them, so a policy that ignores risk will always produce a queue.

Why review by risk?#

When agents write most of the code, reading becomes the scarce resource. Hiring a reviewer takes months; the agents will produce more changes long before then. So the only lever you control is how much reading each change needs, and the size of a diff is the wrong way to decide it. A one-line change to a refund cap is the most dangerous change of the week; a large refactor of a read-only report may be the safest.

Change the policy, step by step#

Step 1 · Measure the queue#

Queue time = review slots needed ÷ review slots available per day. Count slots, not pull requests: a change needing two readers takes two. Measure slots available from reviews actually completed over the last four weeks, not from a job description. At SkyWays, nine changes on a two-senior-reviewer policy needed 18 slots against 4.5 a day: a four-day queue.

Step 2 · Band every change by what it touches#

Give every tool and path a band from the authority budget (R1 read-only, R2 reversible write, R3 hard-to-reverse write, R4 money, R5 irreversible) and let a change inherit the band of the most dangerous thing it touches. No band is ever decided by the size of a diff.

Step 3 · Put the band in a path rule nobody sets for their own work#

Generate a rule file in the repository (a CODEOWNERS file, for example) from the authority budget, so the band of a change is decided by the paths it touches, not by the author. Self-assessed risk is not a control.

Step 4 · Route the readers by band#

At SkyWays routing cut the slots needed from 18 to 7 and the queue from four days to 1.6, with the same people reading at the same speed.

Step 5 · Open the no-reader lane, and count the escapes#

A lane with no reader is exactly as safe as its harness, so count the escapes (defects that reach production through it) every week. Zero escapes over n merges bounds the escape rate at about 3 ÷ n (the rule of three): 3% after 100 merges, 1.5% after 200. Write the lane's charter before opening it, including the condition that closes it, and close it on the first escape.

Step 6 · Report three numbers, monthly, together#

Report review slots needed, days in the queue and escapes from the no-reader lane, and report the escape count even when it is zero. Any one alone can be argued with; together they answer whether you went faster without going blind.

Where you'll use it#

• As an engineering lead, the week the queue passes two days.
• As a programme manager, reading the review queue off the board every week.
• In a security or audit review, where the path rule shows that money changes can never skip two readers.

Why it matters#

Adding reviewers is slow and does not scale with agents. Changing the policy takes an afternoon and cuts most of the queue, while raising the scrutiny on the changes that can actually do harm.

Key takeaways#

1. Reading is the scarce resource: measure the queue in slots, and change the policy, not the headcount.
2. Band by what a change touches, enforced by a path rule: two readers on money, one on writes, none on reads.
3. Count the escapes, and report slots, queue days and escapes together every month.

FAQ#

How do you review AI-generated code?#

By risk. Band each change by the most dangerous tool or path it touches, and route readers by band: two named readers for money, identity or policy; one reader for other writes, after the automated harness has passed; the harness alone for read-only changes, with defects that escape counted weekly.

Should AI-generated code be reviewed differently from human code?#

The same risk rules apply to both. What changes is volume: agents produce more changes than people can read line by line, so a policy that reviews everything equally creates a queue, and a policy that reviews by risk keeps scrutiny where the harm is.

Is it safe to merge code without a human reviewer?#

For read-only, low-risk changes, only as safe as the automated checks that gate the merge, which is why the no-reader lane needs a written charter, a weekly count of escaped defects, and a rule that closes it on the first escape.

What is the rule of three?#

A statistical rule of thumb: if you have seen zero failures in n independent trials, the failure rate is below about 3 ÷ n with 95% confidence. It is how you state what zero escapes through a no-reader lane actually proves.

Apply it in your role#

Across the enterprise. The lane with no human reader needs a written charter and a weekly count of escaped defects across teams. That data is what lets you widen it safely.

The ten-minute workflow. Review routing, generated from what the tools may do:

Here is our authority budget: <tools with risk bands R1 to R5> and our repository layout: <tree>. Write a
CODEOWNERS file that sends R4 and R5 paths to two named owners, R2 and R3 to one, and leaves R1 to the harness.
List any path you could not place, and any file that serves more than one band.

Sources and credits#