Postmortems for AI incidents: find the missing control

In this lesson you'll learn:

• the one question that turns a postmortem from a search for a person into a search for a control;
• how to classify every layer of defence honestly, with evidence;
• the four artefacts that must leave the room for the incident loop to close.

Sound familiar?#

• The action list says "remind staff of the refund policy" and "update the prompt".
• The postmortem ended with a name, and the same incident recurred a month later with a different one.
• Every layer of defence in the design document was "in place", and the incident happened anyway.

A postmortem that ends in a name, a reminder or a prompt edit has changed nothing that a model, or a person under pressure, cannot get past again.

Why ask for the missing control?#

Because people and models both fail, and a system that depends on neither failing will fail. The question who was careless? produces a name; the question which enforced control would have made this impossible? produces a system change. That is the principle of blameless postmortems, and for agentic systems it has a sharper edge: most controls that "failed" were never enforced at all. They were sentences in a prompt.

Run the postmortem, step by step#

Step 1 · Ask the one question, first#

Write it on the wall before anything else: which enforced control, if it had been present, would have made this impossible? Keep bringing the room back to it. The hour has worked if it produces a system, not a name.

Step 2 · Classify every layer#

List every layer of defence the design claimed and mark each one, with evidence:

That was SkyWays on day 82, when a $2,000 refund went out that was not owed: five layers claimed, none enforced, the cap and the approver written only in the prompt. The last column is the one that matters: with either of the two enforced in the tool's signature, the refund is impossible; injection defence and traces change the odds and the visibility, not the outcome. Every "enforced" must have a file, a line that raises, and a test beside it.

Step 3 · Choose the fix that closes the path#

Pick the control that makes the incident impossible rather than less likely (a cap as a typed parameter that raises, a confirmation token only a person's approval can create) and write the test that reproduces the incident. The test of the fix: the incident is now a test that was red an hour ago.

Step 4 · Lower the autonomy level, and say what restores it#

Drop the action one autonomy level (refunds from acting alone to needing an approver) and write the evidence that would restore it. At SkyWays refunds stayed one level down until a fourteen-day shadow run re-earned the level.

Step 5 · Feed it forward#

Four artefacts must leave the room: new golden cases built from the incident (SkyWays added six), an amended decision record, the postmortem record itself, and a brief for the next P0 with a pain, the evidence, the missing control, the fix and its value. Without the brief, the incident loop has not closed. The eight loops

Where you'll use it#

• After every incident, including near-misses caught by a person.
• After every drift alert that reveals a behaviour the design did not intend.
• In a quarterly review, re-reading the layer tables to check that each "enforced" is still enforced.

Why it matters#

A postmortem that ends in a name leaves every other path open, and the incident recurs with a different name. One that ends in an enforced control closes the path for everyone, permanently, and the brief means the next framing starts from what the incident taught.

Key takeaways#

1. One question: which enforced control would have made this impossible, never who was careless.
2. Classify every layer as enforced, a request or absent, with a file and a line for every "enforced".
3. Four artefacts leave the room: golden cases, a decision record, the record, and a brief for the next P0.

FAQ#

How do you run a postmortem for an AI incident?#

Ask which enforced control would have prevented it, classify every claimed layer of defence as enforced, a request or absent, fix the gap with a control in code and a test that reproduces the incident, lower the affected action's autonomy until evidence restores it, and turn the incident into new test cases and a brief for the next cycle.

What is a blameless postmortem?#

A review of an incident that looks for the conditions and missing controls that allowed it rather than for a person to blame, on the principle that people and systems both fail and only system changes prevent recurrence. The practice is described in Google's Site Reliability Engineering.

Why isn't updating the prompt a fix?#

Because a prompt is a request: it lowers the probability of the behaviour but cannot prevent it, and anything the model reads can argue against it. A fix closes the path: a limit in the tool's signature, or an approval the model cannot produce for itself.

Should an AI agent lose autonomy after an incident?#

For the affected action, yes, one level down, with the evidence that would restore it written in the record, such as a clean shadow run over a fixed window. That keeps the system useful while the fix earns trust, and makes restoring the level a decision with evidence rather than a date.

Apply it in your role#

Across the enterprise. Share missing-control findings across teams. The same missing cap is usually sitting in three other agents.

The ten-minute workflow. Classify every layer before anyone names a person:

Here is the incident: <timeline, trace, the action taken>. List every layer that was supposed to stop
it (prompt, tool signature, identity, approval, monitoring) and classify each as enforced, a request,
or absent. Then name the one enforced control that would have made it impossible, and write its test.

Sources and credits#