# P3 Run & Learn: How to Run an AI Agent in Production Safely
Widen on evidence, rehearse the way back, watch the output mix, and report the saving beside the spend, every cycle, before anyone asks.
7 min read · Intermediate · Lesson 7 of 9 in Agentic PDLC fundamentals · Updated 2026-10-02 · By Akash Das
Canonical: https://akash-coded.github.io/aws-bedrock-agentcore-strands/learn/p3-run-and-learn/
> **In short.** P3 Run & Learn widens the agent's share of live traffic only as evidence
> arrives, and rehearses the rollback before it is needed. It watches the output mix for drift, and
> reports what the feature saved **beside** what it cost on one line. It turns every incident, drift
> alert and surprising bill into the brief for the next P0.
**In this lesson** you'll learn:
- how to cut over from shadow to full traffic without betting on a date;
- how to catch drift, the defect with no error message, before a customer does;
- how to report value and cost so the programme survives its first bill.
## Sound familiar?
- Launch day was set in a meeting, and nobody divided the cases you need by the cases you get a day.
- Three months after launch a customer noticed the assistant now offers credits where it used to offer refunds.
- Finance found the token bill before the product manager reported the saving.
P3 is where each of those is either caught by a mechanism or discovered by someone else.
## What is P3 Run & Learn?
P3 is the fourth phase of the [agentic PDLC](https://akash-coded.github.io/aws-bedrock-agentcore-strands/learn/what-is-the-agentic-pdlc/), and the one that
never finishes. The **sponsor** is accountable for it. DevOps runs the flags, traces and recovery;
QA watches drift and runs the injection suite; the product manager reports the two numbers; the
architect turns incidents and bills into design changes.
A P3 cycle ends when **both numbers are reported and the brief for the next P0 exists**.
## P3, step by step
### Step 1 · Widen on evidence, not on a date
Four states, taken one at a time: **shadow**, where the agent decides on real traffic and acts on
nothing; a **5% canary**; **wider**, on evidence; then **all of it**. Each step is a condition, never
a date, and the arithmetic decides how long it takes. At 240 cases a day, 5% is twelve a day, so
five hundred cases takes 42 days. The safe share is the slow one, which is exactly why a cut-over
widens rather than holding.
Money actions stay gated at every stage, whatever the shadow shows. When the SkyWays sponsor had six
weeks instead of ten, the shadow window was cut from fourteen days to seven, on the simple slice
only, with every disagreement read.
### Step 2 · Rehearse the way back, and cap the runaway
Rollback is a capability, and a capability nobody has used is a belief. Before cut-over, throw every
switch with a stopwatch running and write the times down. SkyWays measured four: the kill switch in
**40 seconds**, flag to shadow in **2 minutes**, a prompt rollback in **3**, and a model rollback in
**11**, because it redeploys the runtime. Then cap the runaway: a loop limit on every agent, and a
cost limit per case that ends in a person's queue rather than in another retry.
### Step 3 · Watch for drift
**Drift** is behaviour changing with no deploy, no error and no alert. Your existing monitoring
answers whether the system is up and fast; it was never looking for this. Watch the **output mix** (the share of each kind of decision), because accuracy needs labels and arrives late, while the mix is
visible the same day.
Watch it against **two thresholds**, not one. The SkyWays refund-to-credit mix slid from 61/39 in
week one to 48/52 in week eight, thirteen points, and never tripped its 5% weekly alert, because it
moved 1.9 points a week. A second threshold, on the level against a frozen baseline, fires within
weeks where the weekly one never fires at all. Wire a breach to **re-open the release gate**
automatically.
### Step 4 · Report two numbers on one line
Report what the feature **saved** and what it **cost**, together, every cycle, before anyone asks,
with the two rows that stop either number being gamed: the review hours added, and the re-runs.
A first cycle that saves time and costs more is survivable, if the sponsor hears it from you. On day
ninety SkyWays reported 40 to 45 percent fewer person-days and a token bill of $4,200 on one line,
with the rising review hours beside them and the reason they would fall. The programme continued,
not because the numbers were flattering, but because both of them came from the team.
### Step 5 · Turn what you learned into the next P0
A bill that leaves its estimate is a **design** question, not a finance one. SkyWays' day-75 bill was
4.4 times its estimate on flat traffic, and a per-call log showed four ordinary habits multiplying:
An incident is the same: its postmortem asks which **enforced control** was missing, never who was
careless, and it ends as a typed parameter, a test and new golden cases, plus a brief for the next
P0. [The eight loops](https://akash-coded.github.io/aws-bedrock-agentcore-strands/learn/the-eight-loops/) are how that learning finds its way back.
## Where you'll use it
- **From the first day of live traffic, forever.** P3 does not end; each cycle ends.
- **After every prompt, model or configuration change**, which re-runs the harness and, if the mix
moves, re-opens the release gate.
- **In every sponsor update**, which carries the two numbers on one line.
## Why it matters
Gartner's forecast that over 40% of agentic AI projects will be cancelled by 2027 names escalating
costs as a cause. Costs escalate quietly and are found by finance; drift degrades quietly and is found
by customers. P3 exists so that both are found by you, first, with a design change already drafted.
## Try it
An agent handles 400 cases a day. The team wants 600 cases of live evidence before widening beyond a
10% canary. **How many days will the canary take, and what would halve it?**
Show the answer
**Fifteen days.** 10% of 400 is 40 cases a day, and 600 ÷ 40 = 15. Doubling the share to 20% halves
it to seven and a half days, at the cost of doubling the exposure while the evidence is still
thin. Lowering the requirement would also shorten it, but only honestly if the bar and its lower
bound allow fewer cases. Effort cannot shorten it: evidence arrives at the speed of traffic.