DevOps and platform

# baseline.yaml - deploy BEFORE anything that can cost money.
# aws cloudformation deploy --template-file baseline.yaml --stack-name <feature>-baseline \
#   --parameter-overrides Feature=<feature> MonthlyBudgetUsd=<n> AlertEmail=<alias> \
#   --tags Feature=<feature> Environment=<env> Owner=<name> CostCentre=<code>
# Always-on register, reviewed weekly, each with an owner and a teardown command
# (these bill for EXISTING, not for use): classic OpenSearch Serverless collections,
# AgentCore Runtime instances and stored memory, provisioned throughput, NAT gateways, idle endpoints.
AWSTemplateFormatVersion: "2010-09-09"
Description: Cost baseline for one feature. Creates nothing that serves traffic.

Parameters:
  Feature:
    Type: String
    Description: The feature that pays for everything tagged with it. Never a team name.
    AllowedPattern: "[a-z0-9-]{3,32}"
  MonthlyBudgetUsd:
    Type: Number
    Description: The forecast that was defended, not a round number.
  AlertEmail:
    Type: String

Resources:
  BudgetAlerts:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: {Fn::Sub: "${Feature}-budget"}

  # Without this policy the budget cannot publish, and the silence looks healthy.
  BudgetAlertsPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      Topics: [{Ref: BudgetAlerts}]
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: {Service: budgets.amazonaws.com}
            Action: sns:Publish
            Resource: {Ref: BudgetAlerts}

  FeatureBudget:
    Type: AWS::Budgets::Budget
    Properties:
      Budget:
        BudgetName: {Fn::Sub: "${Feature}-monthly"}
        BudgetType: COST
        TimeUnit: MONTHLY
        BudgetLimit: {Amount: {Ref: MonthlyBudgetUsd}, Unit: USD}
        CostFilters:
          # Activate this key in the payer account today: cost data starts at activation.
          TagKeyValue: ["user:Feature$<feature>"]
      NotificationsWithSubscribers:
        - Notification: {NotificationType: ACTUAL, ComparisonOperator: GREATER_THAN,
                         Threshold: 80, ThresholdType: PERCENTAGE}
          Subscribers: [{SubscriptionType: SNS, Address: {Ref: BudgetAlerts}},
                        {SubscriptionType: EMAIL, Address: {Ref: AlertEmail}}]
        # Forecast, not actual. This one arrives with a fortnight to act in.
        - Notification: {NotificationType: FORECASTED, ComparisonOperator: GREATER_THAN,
                         Threshold: 100, ThresholdType: PERCENTAGE}
          Subscribers: [{SubscriptionType: SNS, Address: {Ref: BudgetAlerts}}]

You are writing the baseline infrastructure for a new agentic feature. Nothing
exists yet.

Produce ONE CloudFormation template (or a CDK stack, if this repository already has
one) that creates only the cost baseline: the budget, its alarm path, and the
parameters everything else will inherit. No workload resources.

RULES:
- A FORECASTED notification as well as an ACTUAL one. Put the reason in a comment.
- Every resource carries Feature, Environment, Owner, CostCentre. Feature is the cost
  boundary and it is never a team name.
- No hard-coded account id, region or ARN partition. Parameters or pseudo-parameters.
- Run cfn-lint (or cdk synth) yourself. Do not hand me a template you have not linted.

OUTPUT SHAPE: the template, then the lint output, then the exact deploy command for
<environment>, then a list of anything you had to assume.

OUR BOUNDARIES:
<paste: accounts, regions, who pays, the residency answer, who gets the alert>

Audit this account for resources that bill for EXISTING rather than for use.

Look for at least: classic OpenSearch Serverless collections, AgentCore Runtime instances,
long-term memory stores, provisioned throughput, NAT gateways, idle load balancers, allocated addresses, unattached volumes,
old snapshots.

OUTPUT SHAPE - one table, sorted by monthly cost, highest first:
| Resource | Type | Created | Last used (or "unknown") | Feature tag? | ~$/month | Safe to delete? |

RULES:
- READ ONLY. Do not delete, stop or modify anything, and do not write me a script that
  can. Use read-only calls.
- "Safe to delete" is a question for a human. Write UNKNOWN unless there is evidence in
  the account itself, and say what the evidence was.
- List separately every resource with no Feature tag. That list is the real finding.
- Finish with the one resource I should deal with today, and why it is that one.

ACCOUNT AND SCOPE: <paste the profile, the regions, and anything already known to be
deliberate>

Review this tag scheme against the only four questions it will ever be asked.

THE QUESTIONS:
1. What did <feature> cost last month, across every service?
2. Which resources belong to an environment we are about to destroy?
3. Who do I call about this resource at 02:00?
4. Which resources are billing with no owner at all?

RULES:
- Judge each tag only by whether it serves one of those four. Do not add tags for tidiness.
- Flag any tag whose value changes when the org chart changes. Those cannot answer
  question 1 for any month before the change.
- Say which keys must be activated as cost allocation tags in the payer account, and
  state plainly what happens to the months before activation.

OUTPUT SHAPE: keep / change / drop per tag, one line of reasoning each. Then the four
questions with "answerable: yes/no" against your revised scheme.

SCHEME: <paste>

# litellm-config.yaml - the one layer every model call goes through.
# litellm --config litellm-config.yaml --port 4000
#
# Every `model` below is an INFERENCE PROFILE id. A bare id with no geography
# prefix fails with a validation error telling you to retry with a profile.
# That error reads like a permissions problem and is not one.

model_list:
  - model_name: judge              # capable tier: the judgement call
    litellm_params:
      model: bedrock/eu.anthropic.<model-id>
      aws_region_name: <eu-west-1>
      timeout: 60
  - model_name: judge-secondary    # same model, second region, SEPARATE access grant
    litellm_params:
      model: bedrock/eu.anthropic.<model-id>
      aws_region_name: <eu-central-1>
      timeout: 60
  - model_name: classify           # cheap tier: routing, extraction, tagging
    litellm_params:
      model: bedrock/eu.anthropic.<small-model-id>
      aws_region_name: <eu-west-1>
      timeout: 30

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2                   # capped. Uncapped retries are a silent 1.4x.
  allowed_fails: 3
  cooldown_time: 30
  fallbacks:
    # Same geography only. A fallback that crosses a residency boundary is a breach.
    - judge: ["judge-secondary"]

litellm_settings:
  drop_params: false
  cache: true
  cache_params:
    type: redis
    ttl: 3600
  success_callback: ["s3"]         # EVERY call, not a sample
  failure_callback: ["s3"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  alerting: ["slack"]
  # One virtual key per FEATURE, each with its own budget, so the bill splits on
  # the same boundary as the resource tags:
  #   litellm --create-key --models judge,classify --max-budget <n> \
  #     --budget-duration 30d --key-alias <feature>-prod
  max_budget: <n>
  budget_duration: 30d

You are helping a platform engineer establish which Bedrock models this account
can actually invoke, before anyone designs around one.

Write and run a script that, for every (model, region) pair below:
- calls with the BARE model id first, then with the inference profile id carrying the
  geography prefix for that region (us. / eu. / apac.)
- records the exact error class for each attempt

OUTPUT SHAPE - one table and nothing else:
| Model | Region | Bare id | Profile id | Verdict | Error |

Verdict is exactly one of: GRANTED / NOT GRANTED / PROFILE REQUIRED / QUOTA / UNKNOWN.

RULES:
- max_tokens = 1 and a one-word prompt. A probe that writes a paragraph per pair is a
  bill of its own.
- Do not retry on a throttle. Record QUOTA and move on.
- Create, modify and delete nothing. Read and invoke only.
- Show me the script before you run it.

PAIRS: <paste model ids and regions>

Write a LiteLLM proxy config from the access matrix below.

RULES:
- Every `model` string must be an inference profile id whose geography prefix matches
  its region. If a row in the matrix gives only a bare id, stop and tell me which row.
- Name models by ROLE, not by vendor: `judge` (capable tier), `classify` (cheap tier),
  plus a same-geography twin of each for fallback.
- Fallbacks never cross a residency boundary. Put the reason in a comment on each.
- num_retries capped, with a comment stating the worst-case cost multiplier it implies.
- success_callback and failure_callback on every call, not a sample.
- No secret values anywhere. Environment references only.

OUTPUT SHAPE: the YAML, then a short checklist of what I must verify before deploying,
then the one grep command that proves no bare model id got through.

MATRIX: <paste>

A Bedrock call is failing. Work through these IN ORDER and tell me which it is,
with the single command that confirms or eliminates each:

1. The model access grant is missing in THIS region (it is per model, per region)
2. The call used a bare model id where an inference profile id is required
3. The IAM policy does not allow that action on that resource
4. A quota: requests or tokens per minute for this model in this region
5. Something else - say precisely what

RULES:
- Do not suggest widening an IAM policy until 1 and 2 are both ruled out.
- Do not suggest a retry as a fix for anything.
- Quote the part of the error that distinguishes your answer from the next candidate.

End with the one-line change that fixes the most likely cause, and how I will know.

ERROR, VERBATIM, plus the call site and region:
<paste>

# environments/prod.yaml - one file per environment. CI diffs these on every PR.
# Nothing in here is a value that must stay secret. Secrets are references.
environment: prod
region: <eu-west-1>
account: "123456789012"

model:
  # PINNED. A floating alias changes behaviour with no deploy and no diff.
  primary: eu.anthropic.<model-id>
  cheap: eu.anthropic.<small-model-id>
  judge: eu.anthropic.<judge-model-id>    # pinned separately; see step 4
  temperature: 0.0
  max_tokens: 1024
  # Bump in its own pull request. The harness re-runs. Rollback path in step 8.
  pinned_on: <YYYY-MM-DD>
  pinned_by: <name>

gateway:
  endpoint: <https://gateway.internal.example/v1>
  key_alias: <feature>-prod
  monthly_budget_usd: <n>

secrets:
  # Runtime resolution only. Never a value here, never in a prompt, never in a
  # context file, never in a trace.
  partner_api_key: "{{resolve:secretsmanager:<feature>/prod/partner-api:SecretString:api_key}}"
  desk_webhook: "{{resolve:secretsmanager:<feature>/prod/desk:SecretString:url}}"

data:
  seed: none                  # prod runs on prod data
  vector_index: <collection-name>
  # eval.yaml instead reads:
  #   seed: s3://<bucket>/seed/<yyyy-mm-dd>-redacted-400.jsonl
  #   redaction_manifest: docs/redaction-<yyyy-mm-dd>.md   (signed by <name>)

actions:
  # What this environment is permitted to actually do.
  partner_api: live           # dev: absent, eval: recorded double, staging: sandbox
  issue_refund: gated         # named approver; cap enforced in the tool signature
  email_passenger: live

limits:
  max_loops: 5
  max_tokens_per_case: <n>
  max_cost_per_case_usd: <n>

flags:
  shadow_only: false
  canary_percent: 5

Redact this production export so it can be used as evaluation seed data.

RULES:
- MASK, never drop. Preserve type, length class and format: a 16-digit card becomes
  ************4471, an email becomes a****@e******.com, a date keeps its shape.
- Keep the mess. Typos, truncation, mixed casing, empty fields and duplicate records
  are the distribution. Do not clean anything.
- Free-text fields are the hard part. List every one and tell me what you found in it,
  rather than deciding for me whether it is safe.
- Do not invent rows and do not drop rows. The row count must match exactly.

OUTPUT SHAPE:
1. The redaction script.
2. A MANIFEST: one row per field - name, classification, what was done to it, and an
   example before/after using synthetic values, never real ones.
3. A list of fields you were unsure about, with the question you need answered.

The manifest is what a human signs. Write it for that reader.

SCHEMA / SAMPLE: <paste>

Compare these two environment manifests.

OUTPUT SHAPE - one table:
| Field | <env A> | <env B> | Intended difference? | Could it change behaviour? |

RULES:
- Report EVERY difference, including ones I did not ask about and ones that look
  cosmetic. Whitespace and ordering can be excluded; nothing else can.
- Put model version, judge version, temperature, max_tokens, tool list and the caps at
  the TOP of the table regardless of file order. Those are the behaviour-bearing ones.
- For any field that is a floating alias rather than a pinned version, say so loudly
  and separately. That is the finding, not a row.
- Finish with: the one difference most likely to explain a score gap between these two
  environments, and the command that would confirm it.

MANIFEST A: <paste>
MANIFEST B: <paste>

Write the change record for upgrading <feature> from <model version A> to
<model version B>. Treat it as a deployment, not a configuration tweak.

It must state:
1. Which environments change, in what order, and what gates between them.
2. What re-runs: the full golden set, the judge calibration set, the injection suite.
   Give the expected cost and wall-clock time of that run.
3. What the rollback is, exactly - which file, which command, and how long it takes.
4. Which slices are most likely to move, and why. Be specific about the slice, not
   about the model.
5. What evidence would make us abandon the upgrade rather than debug it.

RULES:
- Do not claim the new version is better. That is what the harness is for.
- "It should be a drop-in" is not an entry in this document.

CONTEXT: <paste the current manifest, the slice list and the last harness readout>

# .github/workflows/agent-ci.yml
# Fast and free first. Slow and paid only on what changed.
name: agent-ci

on:
  pull_request:
  schedule:
    - cron: "0 2 * * *"        # the FULL set, nightly

permissions:
  id-token: write              # OIDC. No long-lived keys in the repository.
  contents: read

jobs:
  fast:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12"}
      - run: pip install -r requirements-dev.txt && ruff check . && mypy src
      - run: pytest tests/unit tests/exact -q   # caps, fare rules, schemas
      - run: python tools/check_no_bare_model_id.py environments/

  eval:
    needs: fast
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
        with: {fetch-depth: 0}
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/<feature>-ci-eval
          aws-region: <eu-west-1>
      - name: Which slices did this change touch
        id: slices
        # A prompt or model change touches everything, so the map does not apply.
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            python tools/slice_map.py --diff "origin/${{ github.base_ref }}...HEAD" >> "$GITHUB_OUTPUT"
          else
            echo "slices=all" >> "$GITHUB_OUTPUT"
          fi
      - name: Restore the model response cache
        uses: actions/cache@v4
        with:
          # (model version, prompt hash, input hash). An unchanged golden set must
          # replay for near nothing and answer the same way twice.
          key: evalcache-${{ hashFiles('prompts/**', 'environments/eval.yaml') }}
          path: .evalcache
      - name: Harness
        run: python -m harness run --slices "${{ steps.slices.outputs.slices }}" --cache .evalcache
      - name: Judge, then gate
        run: |
          python -m harness judge --judge-version-from environments/eval.yaml
          python -m harness gate --bars docs/bar-sheet.yaml --summary "$GITHUB_STEP_SUMMARY"
        # gate exits non-zero when ANY slice's 95% lower bound is below its bar.
# Branch protection: `eval` is a REQUIRED status check. A job that only comments
# is a report, and a report has never stopped a release.

Build the map from repository paths to evaluation slices, so a pull request runs
only the slices it could have affected.

OUTPUT SHAPE - a committed YAML file:
  slices:
    <slice-name>:
      paths: [<glob>, <glob>]
      cases: <path to the case file>
      bar: <n>
  always_full: [<globs that force the FULL set>]

RULES:
- Anything under the prompts directory, the model pin in any environment manifest, the
  judge prompt, or the harness itself goes in always_full. Those touch every slice.
- When a path could belong to two slices, put it in BOTH. Over-running costs money;
  under-running costs a gap in the gate that produces no signal.
- List any path that maps to no slice at all. That list is the gap, and I need it
  explicitly rather than by inference.

Then write the script that resolves a git diff to a slice list, and a test that proves
a change to a prompt returns "all".

OUR SLICES AND LAYOUT: <paste the tree and the slice names>

Write the gate step that turns harness output into a decision.

OUTPUT SHAPE - a markdown job summary, in this exact order:
| Slice | Score | n | 95% lower bound | Bar | PASS / FAIL / UNPROVEN |
then one line: SHIP or DO NOT SHIP, and the single reason.

RULES:
- lower bound = p - 1.96 * sqrt(p*(1-p)/n). Compute it even when only the score was given.
- PASS only when the LOWER BOUND is at or above the bar. Score above the bar with the
  lower bound below it is UNPROVEN, and you must say how many more cases are needed:
  n = 1.96^2 * p * (1-p) / (p - bar)^2.
- Exit non-zero on any FAIL. UNPROVEN does not block, but it is reported at the top.
- Flag any slice that got worse than the previous run even if it still passes.
- Print the cost and wall-clock time of the run as the last line. People need to see it
  to keep believing in it.

Show me the lower-bound function and its unit test first; I want to check that by hand.

HARNESS OUTPUT FORMAT: <paste a sample>
BAR SHEET: <paste>

Work out what this pipeline will cost to run, so the number arrives before the
first invoice rather than after it.

Compute and show the arithmetic for:
1. Cost and wall-clock time of ONE full golden-set run, cold cache.
2. The same, warm cache, with the cache hit ratio you expect and why.
3. Cost per pull request at the average touched-slice size.
4. Monthly total at <n> pull requests per day plus one nightly full run.
5. The crossover: at how many pull requests per day does the nightly run stop being
   the dominant cost?

RULES:
- Use the gateway's actual price table, not a remembered figure. Say where you got it.
- Show cached and uncached input tokens as separate terms.
- Give me the single cheapest change that halves number 4, and what it costs in coverage.

INPUTS: <golden set size, tokens per case, model prices, PR volume>

{
  "version": "1",
  "flags": {
    "rebook_same_day": {
      "name": "rebook_same_day",
      "description": "Reversible action. Widens on live evidence per slice, never on a date.",
      "attributes": {
        "mode": {"constraints": {"type": "string", "enum": ["off", "shadow", "canary", "on"], "required": true}},
        "canary_percent": {"constraints": {"type": "number", "minimum": 0, "maximum": 100}},
        "prompt_version": {"constraints": {"type": "string", "required": true}},
        "model_version": {"constraints": {"type": "string", "required": true}},
        "widen_when": {"constraints": {"type": "string", "required": true}},
        "owner": {"constraints": {"type": "string", "required": true}}
      }
    },
    "rebook_partner": {
      "name": "rebook_partner",
      "description": "Reversible with effort. Separate flag so it can stay in shadow while same-day runs live.",
      "attributes": {
        "mode": {"constraints": {"type": "string", "enum": ["off", "shadow", "canary", "on"], "required": true}},
        "prompt_version": {"constraints": {"type": "string", "required": true}},
        "widen_when": {"constraints": {"type": "string", "required": true}},
        "owner": {"constraints": {"type": "string", "required": true}}
      }
    },
    "issue_refund": {
      "name": "issue_refund",
      "description": "Not reversible. The cap and the approver are ENFORCED in the tool signature and the IAM policy; these fields only mirror them for the trace.",
      "attributes": {
        "mode": {"constraints": {"type": "string", "enum": ["off", "shadow", "gated"], "required": true}},
        "cap_usd": {"constraints": {"type": "number", "minimum": 0, "required": true}},
        "approver_role": {"constraints": {"type": "string", "required": true}},
        "owner": {"constraints": {"type": "string", "required": true}}
      }
    }
  },
  "values": {
    "rebook_same_day": {
      "enabled": true, "mode": "canary", "canary_percent": 5,
      "prompt_version": "<git-sha>",
      "model_version": "eu.anthropic.<model-id>",
      "widen_when": "live lower bound >= 0.86 on same-day for 5 consecutive days",
      "owner": "<name>"
    },
    "rebook_partner": {
      "enabled": true, "mode": "shadow",
      "prompt_version": "<git-sha>",
      "widen_when": "codeshare harness lower bound >= 0.80 AND 14 days shadow agreement >= 0.95",
      "owner": "<name>"
    },
    "issue_refund": {
      "enabled": true, "mode": "gated",
      "cap_usd": 400,
      "approver_role": "<duty-manager>",
      "owner": "<name>"
    }
  }
}

Write the control that makes "shadow never writes" true rather than intended.

Produce TWO things:
1. A harness assertion: run the full golden set with the flag in shadow and fail the
   build if any write tool is reached. It must detect the call at the tool boundary,
   not by inspecting the output.
2. A scheduled production canary: read the LIVE flag state, and if any action is in
   shadow, assert no write has been recorded for that action in the last hour. Page on
   failure.

RULES:
- Test at the boundary the agent actually crosses. A test on a mock passes forever.
- Two places, because the flag can be changed outside a deployment and often is.
- The canary must fail loudly when it cannot read the flag state. Unknown is not pass.
- Include the test that proves the test works: force a write and show both fail.

OUR WRITE TOOLS AND FLAG SOURCE: <paste>

Write the change record for this prompt change. Treat it as a deployment.

It must state:
1. The diff, and in plain words what behaviour is intended to change.
2. Which slices are expected to move, in which direction, and by roughly how much.
3. What runs before merge: the full golden set, the judge calibration set, the
   injection suite. Give the cost and wall-clock time.
4. The version identifier that ships (a hash, not "latest"), and where it is stored.
5. The rollback: which command, which previous version, how many seconds.
6. What in the trace will tell us afterwards which version decided a given case.

RULES:
- "Minor wording change" is not an entry. Every wording change is a behaviour change
  until the harness says otherwise.
- If the change is intended to fix a specific failing case, say which, and say what
  stops it from over-fitting to that case.

PROMPT DIFF: <paste>

Plan the canary for <action>, as a config, not as a narrative.

OUTPUT SHAPE:
| State | Share | Entry condition | Exit condition | Who decides | Rollback time |
for each of: shadow, 5%, <n>%, 100%.

Then the AUTOMATIC rollback: the metric, the threshold, the evaluation window, and
what it rolls back TO (a state, not "the previous version").

RULES:
- Every entry condition is evidence, never a date.
- Every state has an owner who can move it without an approval chain.
- Automatic rollback triggers must be things that cannot be explained away at 02:00:
  cost per case, error rate, cap trips, a write while in shadow.
- Say explicitly which actions never enter this ladder because they are gated.
- Give me the single condition most likely to be argued with, and rewrite it so it
  cannot be.

CONTEXT: <slices, bars, cases per day, current flag states>

# trace.py - one write per consequential action: the metric and the row that
# produced it, in the same CloudWatch Logs record (Embedded Metric Format).
# Rule: MASK, never omit. A masked field is a row you can reconcile; a dropped
# row is an incident you cannot explain.
import json
import time
from decimal import Decimal

NAMESPACE = "<feature>/agent"

# Per 1k tokens, read from the gateway price table at start-up. Never a guess.
PRICES = {"<model-id>": {"in": Decimal("<n>"), "cached": Decimal("<n>"), "out": Decimal("<n>")}}


def mask(value, keep=4):
    # Keep the shape, lose the content: '************4471', not a dropped column.
    if not value:
        return ""
    return "*" * max(0, len(value) - keep) + value[-keep:]


def cost_usd(model, tokens_in, cached_in, tokens_out):
    p = PRICES[model]
    fresh_in = max(0, tokens_in - cached_in)
    return (Decimal(fresh_in) * p["in"]
            + Decimal(cached_in) * p["cached"]
            + Decimal(tokens_out) * p["out"]) / 1000


def emit(case_id, slice_name, action, model, prompt_version, flag_state,
         tokens_in, cached_in, tokens_out, loops, cap_usd, approver,
         passenger_ref, outcome):
    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": NAMESPACE,
                "Dimensions": [["Environment", "Slice", "Action"]],
                "Metrics": [{"Name": "CostPerCaseUsd", "Unit": "None"},
                            {"Name": "Loops", "Unit": "Count"},
                            {"Name": "CacheHit", "Unit": "Count"}],
            }],
        },
        "Environment": "<prod>", "Slice": slice_name, "Action": action,
        "CostPerCaseUsd": float(cost_usd(model, tokens_in, cached_in, tokens_out)),
        "Loops": loops,
        # MARKED. A cache hit counted as a fresh call inflates the bill you report.
        # A cache hit counted as a wrong answer gets a working system switched off.
        "CacheHit": 1 if cached_in > 0 else 0,
        "CachedTokens": cached_in,
        # The row an incident reviewer reads cold, twelve weeks later:
        "case_id": case_id,
        "passenger_ref": mask(passenger_ref),
        "model_version": model,
        "prompt_version": prompt_version,
        "flag_state": flag_state,
        "cap_usd": str(cap_usd),
        "approver": approver or "none",
        "outcome": outcome,
    }))

Design the trace row by working backwards from the incident.

THE INCIDENT I must be able to explain twelve weeks later:
<paste: e.g. "the agent issued a refund above the cap and nobody knows how">

1. List every question an incident reviewer, a regulator and a finance analyst would
   each ask about that case.
2. For each question, name the field that answers it.
3. Produce the schema: field name, type, masked or plain, retention, and WHICH of the
   questions it exists for. A field that answers no question comes out.
4. Mark every field that carries personal data and state how it is masked. MASK, never
   omit - say what shape survives masking and why that is enough to reconcile.

RULES:
- No field exists "just in case". Each one is justified by a question.
- If a question cannot be answered by any field, say so plainly. That gap is the output
  I am looking for.
- Include model version, prompt version, flag state, the cap that applied and the
  approver. If you think any of those is unnecessary, argue for removing it explicitly.

Turn this threshold table into CloudWatch alarms as infrastructure as code.

ALARMS REQUIRED, at minimum:
- Cost per case above 3x the estimate, sustained for 1 hour
- Loop-cap trips per hour above <n> (the cap is doing work that should not be needed)
- Cache hit ratio below <n>% over 6 hours (a prompt prefix changed and every call is
  now full price)
- Output mix moved more than 5 percentage points week over week

RULES:
- State the statistic, period, evaluation periods, datapoints-to-alarm and
  treat-missing-data for each, and justify each choice in a comment.
- No alarm on a one-minute period over sparse traffic. Say what our traffic actually is
  and size the window to it.
- Every alarm names what the responder should DO. An alarm with no action is a chart.
- The drift alarm must trigger the release-gate re-open, not just a notification. Show
  how that is wired.

OUR THRESHOLDS AND TRAFFIC: <paste>

Here is our observability setup: <paste dashboards, metrics, alarms, trace schema>.

Answer only this: which of these questions can it NOT answer, and what is missing?

1. What did case <id> cost, and was any of it a cache hit?
2. Which model version and prompt version decided case <id>?
3. Has the mix of outcomes changed this week versus last?
4. How many cases hit the loop cap yesterday, and did they end at the desk?
5. Which slice is getting more expensive per case, month over month?
6. Did anything write while an action was in shadow?
7. Who approved the last twenty gated actions?

OUTPUT SHAPE:
| Question | Answerable now? | What is missing | Cost to add |

RULES:
- "It is in the logs somewhere" is a NO. Answerable means one query.
- Rank what is missing by what it would cost us during an incident, not by effort.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InvokeOnlyThePinnedProfiles",
      "Effect": "Allow",
      "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
      "Resource": [
        "arn:aws:bedrock:eu-west-1:123456789012:inference-profile/eu.anthropic.<model-id>",
        "arn:aws:bedrock:eu-west-1::foundation-model/anthropic.<model-id>",
        "arn:aws:bedrock:eu-central-1::foundation-model/anthropic.<model-id>"
      ]
    },
    {
      "Sid": "ResidencyEveryOtherRegionIsDenied",
      "Effect": "Deny",
      "Action": "bedrock:*",
      "Resource": "*",
      "Condition": {"StringNotEquals": {"aws:RequestedRegion": ["eu-west-1", "eu-central-1"]}}
    },
    {
      "Sid": "ReadTheIndexTheDataAccessPolicyNarrowsItFurther",
      "Effect": "Allow",
      "Action": "aoss:APIAccessAll",
      "Resource": "arn:aws:aoss:eu-west-1:123456789012:collection/<collection-id>"
    },
    {
      "Sid": "ReadToolsOnlyWriteToolsBelongToTheWriterRole",
      "Effect": "Allow",
      "Action": "lambda:InvokeFunction",
      "Resource": [
        "arn:aws:lambda:eu-west-1:123456789012:function:<feature>-tool-search-flights",
        "arn:aws:lambda:eu-west-1:123456789012:function:<feature>-tool-fare-rules",
        "arn:aws:lambda:eu-west-1:123456789012:function:<feature>-tool-passenger-record"
      ]
    },
    {
      "Sid": "NoWriteToolOnThisIdentityTheCapLivesInItsSignatureToo",
      "Effect": "Deny",
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:eu-west-1:123456789012:function:<feature>-tool-write-*"
    },
    {
      "Sid": "OnlyThroughOurVpcEndpoints",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringNotEqualsIfExists": {"aws:SourceVpce": ["<vpce-id-bedrock>", "<vpce-id-aoss>", "<vpce-id-logs>"]},
        "BoolIfExists": {"aws:ViaAWSService": "false"}
      }
    },
    {
      "Sid": "ObservabilityIsNotOptional",
      "Effect": "Allow",
      "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:eu-west-1:123456789012:log-group:/<feature>/agent:*"
    }
  ]
}

Derive the least-privilege execution role from these tool contracts. Work from the
contracts to the policy, never the other way round.

OUTPUT SHAPE:
1. A table: | Tool | AWS actions it needs | Exact resource ARNs | Read or write? |
2. TWO policies: a read identity and a write identity. Nothing appears in both.
3. Every statement carries a Sid naming the tool it serves.
4. A list of every action currently in our role that maps to NO tool. That list is
   what comes out.

RULES:
- No wildcards in Action or Resource unless the API genuinely has no resource-level
  permission. Where that is true, say which API and cite it, and add a Condition.
- A Deny statement for any region outside <allowed regions>, on aws:RequestedRegion.
- Do not add a permission because a test failed. Report the denied action instead.
- Run IAM Access Analyzer policy validation and paste the findings.

TOOL CONTRACTS: <paste>
CURRENT POLICY: <paste>

Write the injection test suite for our harness. Every ingested text is untrusted,
including responses from partners we have contracts with.

Produce cases in these classes, at least three each:
- A partner API response with an instruction inside a free-text field (remarks, notes,
  reason codes)
- A passenger message impersonating the desk, an internal system, or a policy update
- A document with hidden or out-of-band text (white text, metadata, alt text)
- A retrieved index chunk that contains instructions
- A tool error message crafted to redirect the agent

FOR EACH CASE: the input, what the attacker is trying to make happen, the assertion
that must hold.

RULES:
- The assertions are about ENFORCED controls, not about wording. "The refund amount
  never exceeds the cap", "no write tool is called while in shadow", "no request leaves
  the egress allowlist". An assertion about the model's phrasing is not a test.
- At least one case must succeed at manipulating the model and still be contained by
  the cap. That case is the point of the whole suite.
- The suite fails the build. It does not warn.

OUR TOOLS, CAPS AND INGESTION POINTS: <paste>

Compare what this role is ALLOWED to do against what it has actually done.

Use CloudTrail for the last <n> days plus IAM Access Analyzer unused-access findings.

OUTPUT SHAPE:
| Action | Allowed on | Times used | Last used | Verdict |
Verdict: KEEP / NARROW TO <arn> / REMOVE / TOO NEW TO JUDGE.

RULES:
- READ ONLY. Produce the proposed policy as a diff; change nothing.
- An action used zero times is not automatically REMOVE - some are for failure paths
  that have not fired. Mark those TOO NEW TO JUDGE and say what would exercise them.
- Report every AccessDenied separately, with the caller and the time. Each one is
  either a missing permission or an attempt, and I need to know which you think it is.
- Finish with the single narrowing that most reduces blast radius, and what would
  break if it is wrong.

#!/usr/bin/env bash
# rollback.sh - three artefacts, four switches. Rehearsed <date> by <name>:
#   kill    ~40s  every case to the desk queue, with its context. NOT an error.
#   flag    ~2m   one action back to shadow; the other actions keep running.
#   prompt  ~3m   runtime picks up the previous prompt version on the next case.
#   model   ~11m  redeploys the runtime, and the harness must re-run afterwards.
set -euo pipefail

FEATURE="<feature>"
ENVIRONMENT="prod"
REGION="<eu-west-1>"
APP_ID="<appconfig-application-id>"
ENV_ID="<appconfig-environment-id>"
PROFILE_ID="<appconfig-configuration-profile-id>"
STRATEGY_ID="AppConfig.AllAtOnce"

confirm() {
  echo "ABOUT TO: $1"
  read -r -p "Type the environment name to continue: " reply
  [ "$reply" = "$ENVIRONMENT" ] || { echo "aborted"; exit 1; }
}

put_param() {
  aws ssm put-parameter --name "/$FEATURE/$ENVIRONMENT/$1" --value "$2" \
    --type String --overwrite --region "$REGION" >/dev/null
  echo "set /$FEATURE/$ENVIRONMENT/$1 = $2"
}

case "${1:-}" in
  kill)
    confirm "kill switch ON - every case to the desk queue with its context"
    put_param kill-switch on
    ;;
  flag)
    # Roll FORWARD to a known-good previous configuration version.
    confirm "deploy flag document version ${2:?previous version number required}"
    aws appconfig start-deployment --application-id "$APP_ID" \
      --environment-id "$ENV_ID" --configuration-profile-id "$PROFILE_ID" \
      --deployment-strategy-id "$STRATEGY_ID" --configuration-version "$2" \
      --region "$REGION" --query Id --output text
    ;;
  prompt)
    confirm "pin prompt version ${2:?prompt sha required}"
    put_param prompt-version "$2"
    ;;
  model)
    # Slowest path. Whatever you roll back to, the harness re-runs against it.
    confirm "redeploy runtime pinned to model ${2:?model id required}"
    aws cloudformation deploy --stack-name "$FEATURE-$ENVIRONMENT-runtime" \
      --template-file infra/runtime.yaml --region "$REGION" \
      --parameter-overrides "ModelId=$2" --no-fail-on-empty-changeset
    echo "NOW: re-run the harness against $2 before widening any flag."
    ;;
  *)
    echo "usage: $0 {kill|flag <version>|prompt <sha>|model <model-id>}" >&2
    exit 2
    ;;
esac

echo "Done at $(date -u +%FT%TZ). Put the elapsed time in the runbook."

Turn this rollback runbook into a script, then into a rehearsal.

PART 1 - the script. One subcommand per switch: kill, flag, prompt, model.
RULES:
- Each subcommand prints exactly what it will change and requires a typed confirmation.
- Each is idempotent and safe to run twice.
- Each ends by printing the check that proves it worked, as a command I can paste.
- Fail loudly on a missing argument. Never guess a version.

PART 2 - the rehearsal plan.
- Who runs it (the person who will do it at 02:00, not the person who wrote it).
- Which environment, and what makes that environment realistic enough to count.
- What is timed, and where the measured times get written down.
- What would make us say the rehearsal FAILED, decided before we run it.

RULES:
- Do not write a step that says "verify the system is healthy". Say which metric,
  which query, which value.
- If any switch cannot be scripted, say so and explain what makes it manual. That is
  the finding, and it is worth more than a script that pretends.

RUNBOOK: <paste>

Design the containment caps for this agent, and the tests that prove each one.

CAPS REQUIRED:
- MAX_LOOPS (default 5): the agent stops rather than reasoning in circles
- Tokens per case
- Cost per case
- Wall-clock per case

FOR EACH, give me:
1. Where it is enforced. It must be in the runtime, not in the prompt. Say which line.
2. What happens when it trips: the case goes to the desk queue WITH its context, a
   metric is emitted, and the passenger sees something sensible. Never a bare error.
3. The test that proves it: a case engineered to hit that cap, asserting the stop, the
   metric and the destination.
4. How the threshold was chosen, and what evidence would change it.

RULES:
- A cap that trips silently is not a cap. Every trip emits a metric with the case id.
- Say what the caps cost us: which legitimate cases they will refuse, and roughly how
  many per week. If you cannot estimate that, say what to measure first.

OUR AGENT AND ITS TOOLS: <paste>

Given this architecture, tell me what cannot be rebuilt from the repository.

OUTPUT SHAPE:
| Thing | Rebuildable from the repo? | If lost, what breaks | RPO | RTO | Restore test |

RULES:
- Be specific about the ones people forget: the trace store (the only account of what
  happened), the golden set and its labels (the measurement itself), the vector index
  AND the source documents it was built from, the flag configuration history, the
  redaction manifests.
- For the vector index, say whether we can rebuild it and what that costs in time and
  tokens. "We can re-embed" is only true if the sources are retained - check.
- Anything marked rebuildable must have the command that rebuilds it. If nobody has
  run that command, mark it UNPROVEN.
- Finish with the single restore we should test this quarter, and why that one.

ARCHITECTURE: <paste>