Lab 1 · Grow the spec · the replies behind its debrief
Three more models, the same two prompts
We ran two of this lab's prompts through three more models on 2 October 2026, word for word. Claude's replies are the lab's own recordings.
| Priya's page said | Claude | Kimi K3 | GLM-5 | DeepSeek V3.2 |
|---|---|---|---|---|
| "much faster"the wait for a decision | Under 5 minutes | Under 10 minutes | Under 10 minutes | Under 5 minutes |
| "Most cases need no agent" | 60% | 70% or more | Over 70% | Over 70% |
| "Mistakes are rare"the error rate | Under 1% | Under 1% | Under 1% | Under 2% |
| "no sensible flight"when a refund is offered | None arriving within 24 hours | Left to Ops to define | No same-day flight; a delay over 4 hours | None within a 24-hour window |
| "a reasonable limit"the refund cap | Left to Finance | Left to Finance | Left to Finance | Left to Finance |
| Marked as proposed | Each one it filled in | A note above its targets | Nothing | Nothing; marked Final |
| In the spec | Claude | Kimi K3 | GLM-5 | DeepSeek V3.2 |
|---|---|---|---|---|
| Times it wrote NOT DECIDED | 11 | 8 | 5 | 6 |
| Field 4, the model's rolenot on the PRD's list | NOT DECIDED | NOT DECIDED | NOT DECIDED | NOT DECIDED |
| The PRD's three open questionsthe bar, the records, the fare engine | All three, with their owners | All three, with their owners | All three, with their owners | All three, with their owners |
| A refund up to $400may the assistant act alone? | NOT DECIDED | NOT DECIDED | NOT DECIDED | May act alone |
The numbers differ. The places do not. Told where to stop, all four wrote NOT DECIDED at least once in each of the five fields, and one still let the assistant act alone on a refund up to $400.
The replies, as the models wrote them
Each prompt was the whole message.
Prompt AWrite the full PRD from it
Below is a one-page PRD-lite for a rebooking assistant at an airline. Write the full PRD from it. Keep it to one page.
[ PRD-lite · Rebooking assistant, in full ]
Recorded reply · Kimi K3 · Moonshot AI · 2 October 2026 · to Prompt A
Recorded reply · GLM-5 · Z.ai · 2 October 2026 · to Prompt A
Recorded reply · DeepSeek V3.2 · DeepSeek · 2 October 2026 · to Prompt A
Prompt BWrite NOT DECIDED and who decides
Below is a PRD. Turn it into a one-screen spec with exactly these eight fields: 1 Title 2 Value 3 Acceptance criteria 4 The model's role (which steps a model decides, which are exact code) 5 Autonomy (for each action: may it act alone, or who approves) 6 The bar (how often it must be right, for each kind of case) 7 Fallback (what happens when it cannot decide or a tool fails) 8 Records (what is written down for each action)
For fields 4 to 8, where the PRD does not say, write NOT DECIDED: followed by the question that must be answered and who should answer it. Do not infer and do not fill in a sensible default. Those gaps are what I am looking for.
PRD: [ PRD · Rebooking assistant, v1, in full ]
Recorded reply · Kimi K3 · Moonshot AI · 2 October 2026 · to Prompt B
Recorded reply · GLM-5 · Z.ai · 2 October 2026 · to Prompt B
Recorded reply · DeepSeek V3.2 · DeepSeek · 2 October 2026 · to Prompt B
The lab's own recordings, for the first column
Recorded reply · Claude · 2 October 2026 · to Prompt A
Recorded reply · Claude · 2 October 2026 · to Prompt B