Mental models

Twelve rules of thumb for software that decides

A procedure tells you what to do on Tuesday. A rule of thumb tells you what to expect before you start, so you can make a good call on a case this manual never covered.

How each card is built

In plain words: the idea, with no jargon
At SkyWays: the case where it bit
What it predicts, and the mistake it prevents
The subtlety: where it stops applying
You have it when: the test

The bar down each card's left edge is the cost of ignoring the rule, yellow to red. All twelve on one page: the poster.

Who this page is for, and how to use it

For

Anyone who has to make a call this manual never covered.

Use it to

Predict what will happen before you start, and name the mistake before it is made.

How

Skim the twelve tiles and jump to one.
Read the card top to bottom: plain words, the SkyWays case, what it predicts, the mistake, the subtlety.
Use the last line, You have it when, as the test.

Twelve rules, each with the mistake it prevents

Keep every chain of model steps shortChained probabilistic steps multiply. They do not average.
Gate by reversibility, not by accuracyWhat a mistake costs matters less than whether you can undo it.
Use a human hold to lower the bar, not to slow the linePutting a person in the loop lowers the damage, and therefore lowers the accuracy you need.
Put every hard limit in code, not in the promptA rule the model reads lowers a probability. A rule the code enforces closes a path.
Read quality per slice, never as one averageAggregate quality is dominated by the easy, high-volume cases.
Quote every score with its sample size and its lower boundA measurement from a sample is an estimate with a width, and the width is the argument.
Plan a cut-over by cases per day, not by the calendarYou cannot learn faster than your sample accumulates.
Decompose the bill into its four habits before you fix anythingA bill is four ordinary behaviours multiplying, not one runaway.
Start with one agent, and add another only for a named limitThings happening at once does not mean several agents.
Set the depth of process per change, not per programmeRun only the lifecycle stages this particular change actually needs.
End a phase on an artefact, never on a dateA hand-off happens when the next person has what they cannot start without.
Watch the output mix, because drift raises no errorA probabilistic system changes behaviour when the world moves, with no deploy.

Cost of ignoring it: high1 of 12

Keep every chain of model steps short

Chained probabilistic steps multiply. They do not average.

In plain words: Every model call is right only most of the time. Chain calls and the chances multiply: two steps at 90 percent are right 81 percent of the time, four are right 66 percent of the time. A chain does not average its steps; it compounds their errors.
At SkyWays: The rebooking assistant ran four model steps in a row: read the disruption, find the fare rule, pick the options, write the message. Each tested at about 90 percent. End to end it was right two times in three, and every wrong answer read as confidently as a right one.
What it predicts: Four steps each right 90% of the time are right 66% of the time end to end, and they fail fluently: no exception, no red test, a confident wrong answer. Every step you add is a tax on every step before it.
The mistake it prevents: Judging a pipeline by its weakest step, or by the mean of its steps. Both readings are optimistic, and the second is the one that gets written in a status report.
The subtlety: It is the optimistic bound, because real steps correlate: a bad retrieval makes the next three worse. If your measured end-to-end rate is below p to the n, correlation is why, and the fix is upstream of the step you were blaming.
You have it when: You reach for a multiplication before you reach for an average, and your first instinct on a long chain is to remove a step rather than improve one.

Where it does its work The arithmetic · Where the checkers go

Cost of ignoring it: severe2 of 12

Gate by reversibility, not by accuracy

What a mistake costs matters less than whether you can undo it.

In plain words: A two-way door is a decision you can walk back cheaply: a draft, a proposal, a suggested list. A one-way door cannot be undone: a refund paid, an email sent, a booking cancelled. Two actions with the same accuracy need different controls when one is a two-way door and the other is one-way.
At SkyWays: Proposing three rebooking options is a two-way door: the passenger picks, and a poor option costs nothing. Paying the refund is a one-way door: once paid, it is gone. SkyWays let the model propose on its own and put a person on every refund, whatever the accuracy figure said.
What it predicts: Two actions with the same expected loss need different controls if one can be withdrawn and the other cannot. A proposal can be retracted; a cash refund cannot. The second needs a person regardless of how accurate the model becomes.
The mistake it prevents: Setting autonomy from model capability, and setting it per product. Both produce a level that is simultaneously too loose for the money action and too strict for the harmless one.
The subtlety: Reversibility is a property of your business, not of the software. The same refund is a two-way door at a company that can claw back and a one-way door at one that cannot, and no amount of engineering changes which you are.
You have it when: Your first question about a new action is “can we undo it, and how fast”, before anyone has mentioned accuracy.

Where it does its work Autonomy, per action · The risk ladder · Decide it for one action

Cost of ignoring it: moderate3 of 12

Use a human hold to lower the bar, not to slow the line

Putting a person in the loop lowers the damage, and therefore lowers the accuracy you need.

In plain words: A hold is a person checking the model's work before it takes effect. It does not slow the line; it lowers the damage a mistake can do. Because the acceptance bar is derived from damage, a hold also lowers the accuracy you need before you can ship.
At SkyWays: A refund step with $600 of damage per mistake needed 98 percent accuracy to break even. With an agent checking the charge before it went out, the damage fell to $30 and the same step needed 71 percent. Nothing about the model changed; the arithmetic did.
What it predicts: A refund with $600 of damage needs 98% accuracy to break even. Put a person on the charge, drop the damage to $30, and the same step needs 71%. Nothing about the model changed; the arithmetic did.
The mistake it prevents: Treating a human check as friction to be removed, and treating an unreachable bar as a reason not to build. Both follow from reading the hold as a brake.
The subtlety: This is why a gate is a commercial instrument. It is the cheapest way to make a feature shippable, and the case for removing one is an argument about damage, never about speed.
You have it when: When somebody says the accuracy is not good enough, you ask what a mistake costs before you ask how to improve the model.

Where it does its work Derive the bar · Try the arithmetic

Cost of ignoring it: severe4 of 12

Put every hard limit in code, not in the prompt

A rule the model reads lowers a probability. A rule the code enforces closes a path.

In plain words: A rule written in the prompt is a request: the model usually follows it, but text arriving from anywhere can talk it out of it. A rule written into the tool's signature, as a typed parameter that raises, is a boundary nothing can argue with. Keep both: the prompt for good behaviour by default, the code for the day the prompt is talked past.
At SkyWays: The first refund cap lived in a prompt: never refund more than $400. A pasted customer email carrying new instructions got past it in testing. The cap moved into the refund tool's signature, where any value over $400 raises and the call stops.
What it predicts: Every limit written in prose will eventually be crossed, because a model can be talked past a request and text arriving from anywhere can do the talking. A typed parameter that raises cannot be argued with.
The mistake it prevents: The most expensive sentence in agentic software: “we have a cap”, said about a cap that lives in a prompt. Nothing in a code review flags it, because it reads exactly like a rule.
The subtlety: Both belong. The prompt explains the rule so the agent behaves well by default; the signature makes bad behaviour impossible when the prompt has been talked past. “Put it in the tool, not the prompt” is half right and throws away the half that makes the agent cooperative.
You have it when: When told a control exists, you ask to be shown it, and you notice whether somebody opens prose or code.

Where it does its work The six controls · In code · Map the control per tool

Cost of ignoring it: high5 of 12

Read quality per slice, never as one average

Aggregate quality is dominated by the easy, high-volume cases.

In plain words: One overall score is dominated by the easy, high-volume cases. The slice that carries the risk can fall below its bar while the average rises. Read quality per slice, each slice against its own bar.
At SkyWays: The overall score rose from 79 to 84 percent in one release. The codeshare slice, the one with the money in it, fell from 81 to 77 against a bar of 80. The headline number endorsed a regression.
What it predicts: An overall score can rise while the slice carrying all the risk falls below its bar. It happened at SkyWays: 79% to 84% overall, and codeshare down from 81% to 77% against a bar of 80.
The mistake it prevents: Shipping a regression that the headline number endorses, and setting one bar for a whole feature so the hard slice ships broken while the easy one waits.
The subtlety: The same trap runs through sampling. A sample representative of traffic is not representative of risk, so the rare costly slice needs deliberate oversampling rather than a bigger random draw.
You have it when: You ask “per slice?” before you read any quality number, and a single percentage in a deck makes you suspicious rather than reassured.

Where it does its work Per-slice readouts · Rare slices

Cost of ignoring it: high6 of 12

Quote every score with its sample size and its lower bound

A measurement from a sample is an estimate with a width, and the width is the argument.

In plain words: A score from a sample is an estimate with a width. Forty cases at 82 percent could really be 70 percent; five hundred cases at 82 percent are at least 79 percent. Only the lower bound can prove a bar, and proving a score that sits just above the bar takes far more cases than proving one well above it.
At SkyWays: QA reported 82 percent on forty codeshare cases against a bar of 80. The lower bound was near 70, so nothing was proven. They owed another 460 cases and said so, which turned an argument into a plan.
What it predicts: 82% on forty cases and 82% on five hundred are different claims. The first has a lower bound near 70%, the second near 79%. Against an 80% bar, neither is proven. And no realistic sample will prove it, because the estimate sits too close.
The mistake it prevents: Shipping on a point estimate, and rejecting a slice that is merely unproven. The second matters: “not proven” with a cases-owed number is a plan, where “it failed” is an argument.
The subtlety: The cost of proving grows quadratically as your score approaches the bar. Halve the gap and you quadruple the cases. A score a whisker above the bar is the most expensive result you can get.
You have it when: You never quote a score without its sample size, and you hear “94% accurate” as an incomplete sentence.

Where it does its work Lower bounds · The calculator

Cost of ignoring it: moderate7 of 12

Plan a cut-over by cases per day, not by the calendar

You cannot learn faster than your sample accumulates.

In plain words: Production evidence arrives at the rate of the traffic you expose. At five percent of 240 cases a day you see twelve cases a day, so five hundred cases takes forty-two days whatever the deadline says. A cut-over is a schedule for widening exposure, not a date.
At SkyWays: SkyWays wanted a launch date. Dividing the cases needed by the cases per day gave forty-two days at five percent, so they widened to fifteen percent after two clean weeks and reached the number in twenty-four days.
What it predicts: At 240 cases a day and five percent of traffic you see twelve cases a day, so five hundred cases takes forty-two days. That number is fixed by arithmetic, not by effort, and no amount of urgency moves it.
The mistake it prevents: Promising a cut-over date before anyone has divided cases-needed by cases-per-day, and sitting at five percent indefinitely because it feels safe.
The subtlety: The safe share is the slow one, which is the whole reason a cut-over widens rather than holding. And the sample is biased by time: at five percent for six weeks you meet a normal Tuesday many times and a storm day perhaps once, so widen across conditions rather than only across volume.
You have it when: When asked for a launch date you reach for a division, and you can say what the window buys as well as what it costs.

Where it does its work Cut over and widen · The arithmetic · Days per traffic share

Cost of ignoring it: high8 of 12

Decompose the bill into its four habits before you fix anything

A bill is four ordinary behaviours multiplying, not one runaway.

In plain words: A surprising bill is rarely one runaway. It is four ordinary habits multiplying: a bloated context, no routing to cheaper models, a cache that never hits, and retries. Each looks sensible alone. Fix the habit that removes the most multiplier per day of work.
At SkyWays: The bill came in at 4.4 times the estimate on flat traffic: context 1.6, no routing 1.5, a discarded cache 1.3, retries 1.4. The retry breaker felt most urgent and was the smallest factor of the four.
What it predicts: Context bloat 1.6, no routing 1.5, a discarded cache 1.3, extra attempts 1.4: and the invoice is 4.4 times its estimate on flat traffic. Each decision was sensible and made by a careful person.
The mistake it prevents: Hunting for the one thing that broke, and fixing the biggest ratio change first. Attempts rose more than five-fold in relative terms and contribute the smallest factor of the four.
The subtlety: Because they multiply, the right fix order is what removes the most multiplier per day of work, (factor − 1) ÷ days, which is usually not the fix that feels most urgent. The retry breaker is the right fix in the wrong position.
You have it when: A surprise invoice makes you open the per-call log rather than the price list, and you expect to find four things rather than one.

Where it does its work Decompose a bill · In the platform · Find the leak

Cost of ignoring it: moderate9 of 12

Start with one agent, and add another only for a named limit

Things happening at once does not mean several agents.

In plain words: Work that happens at once does not need several agents. One agent with a tool that fans out can search four partners in parallel with no hand-offs. Five agents have ten possible hand-offs, and the coordination grows faster than the work.
At SkyWays: The first design had five agents: one per partner airline plus a coordinator. It became one agent with a fan-out search tool. Same parallelism, zero hand-offs, and a calculator in code where a pricing agent had been guessing.
What it predicts: Five agents have ten possible hand-offs, and coordination cost grows faster than the work. One agent with a fan-out tool searches four partners in parallel with zero hand-offs and nothing to get wrong between them.
The mistake it prevents: The reflex that turns “these run concurrently” into “these need separate agents”, which is how a swarm arrives without anyone choosing one.
The subtlety: The rule is start single and escalate on a named limit written into the record: a context that genuinely overloads, or parallel sub-tasks a tool cannot express. Without the written limit the swarm returns by default at the next design review, because nobody can point at what was decided.
You have it when: You ask what limit justifies each hand-off, and you notice that a calculator behind an agent is a provable step made probabilistic.

Where it does its work How many agents · On paper

Cost of ignoring it: moderate10 of 12

Set the depth of process per change, not per programme

Run only the lifecycle stages this particular change actually needs.

In plain words: Not every change deserves the whole process. A one-line fix and a new subsystem need different amounts of discovery, records and gates. The architect sets the depth per change: the spec stays everywhere, and everything around it flexes.
At SkyWays: Eleven gates went on a printer-helpdesk question, and the reputation that earned got the gates skipped on the refund tool, where they mattered. After that, every change was classified before a process was chosen for it.
What it predicts: One process for everything over-serves the one-line fix and under-serves the new subsystem. Both failures are expensive, and the first is the one that gives the method a reputation for slowing teams down.
The mistake it prevents: Eleven gates on a printer-helpdesk question, and the reputation that earns, which is then used to skip the gates on the refund tool, where they mattered.
The subtlety: The spec stays everywhere; it is the backbone. What flexes is everything around it: the document trail, the depth of discovery, the number of records. And the judgement is per change, made by the architect, not per programme set by a policy.
You have it when: You classify a change before you choose a process for it, and you are comfortable saying that a piece of work deserves almost none of this.

Where it does its work Depth per change · The helpdesk case

Cost of ignoring it: moderate11 of 12

End a phase on an artefact, never on a date

A hand-off happens when the next person has what they cannot start without.

In plain words: A phase is finished when the next person holds what they cannot start without: a signed spec, a bar sheet, a report. A date hands over nothing. Only one hand-off is hard; the others may cross with a placeholder, a named owner and a date.
At SkyWays: A spec was signed off in a meeting with five of its eight fields undecided. The engineer decided them by default, because the code had to do something. Three weeks later the missing decision was rediscovered, as the wrong one.
What it predicts: Phases that end on dates hand over nothing, and the receiving team rediscovers the missing decision three weeks later, usually the one nobody wanted to make.
The mistake it prevents: A spec signed off in a meeting with five of its eight fields undecided, which the engineer then decides by default because the code has to do something.
The subtlety: Only one of the four hand-offs is a hard gate. The other three can cross with a placeholder, a named owner and a date, which is what keeps velocity while the decision is still being measured. Treating all four as hard is its own failure.
You have it when: You ask what crossed rather than whether the phase finished, and an empty evidence line reads to you as a blocked merge.

Where it does its work What crosses each hand-off · Hard and soft gates · What a complete pack holds

Cost of ignoring it: high12 of 12

Watch the output mix, because drift raises no error

A probabilistic system changes behaviour when the world moves, with no deploy.

In plain words: A probabilistic system changes behaviour when the world changes, with no deploy and no error. Accuracy needs labels and arrives late, so watch the mix of outputs instead, week on week and against a frozen baseline.
At SkyWays: Nothing changed in the code for three months. A customer noticed the assistant now offered credits where it used to offer refunds. No alert had fired, because nothing was watching the mix of outcomes.
What it predicts: No code changed, nothing threw, no alert fired, and three months later a customer notices the assistant offers credits where it used to offer refunds. Your existing monitoring was never looking for this.
The mistake it prevents: Believing that “nothing changed” means nothing changed, and treating post-launch quality as a testing problem rather than an operational one.
The subtlety: Watch the output mix, not the accuracy: accuracy needs labels and arrives late. And watch two thresholds: the week-on-week step, and the level against a frozen baseline, because a slide of two points a week never trips a five percent rule and still moves you twenty-six points in a quarter.
You have it when: You treat an output distribution as a business metric, and you know what automatically re-opens your release gate.

Where it does its work Watch for drift · As a KPI

How to make them stick

Teach one a week, not twelve at once.A model lands when somebody uses it unprompted in an argument. That takes a fortnight of the same idea being available, not an hour of all of them.
Use the landed-when line as the test.Not whether the team can define it. Whether the question it implies has started showing up in reviews.
Expect three to be resisted.Usually the hold as a lever, the average hiding the slice, and depth as a dial, because each one contradicts something a competent person currently believes is good practice.
Pair each with its arithmetic once.The intuition is what you carry; the formula is what settles the argument. A model without its number loses to a confident opinion.
Keep the subtlety visible.Every one of these has a boundary, and a model applied past its boundary does more damage than no model, because it comes with confidence.

Where they came from. Some are established ideas applied to a new setting: one-way doors, least privilege, Little's law. Some are this manual's own constructions and are defaults to argue with rather than findings. Which is which is recorded on Sources and confidence.

Each rule points at the steps where it does its work.

Walk a role, step by step The lifecycle they belong to