AI product manager interview questions, with frameworks

• Agree with the goal, change the unit. The goal is value from AI, not a count of agents; propose outcome targets (time saved and cost per case) instead.
• Run AI-fit across the candidates: a genuine judgement call, enough volume, a recoverable mistake. Expect most to come back as rules or as assisted features, and publish the rejected list: it is the most credible artefact you will circulate all year.
• Offer the two or three that pass, each with a bar per slice and autonomy per action, and a first feature chosen for provability.
• The insight: an agent-first directive makes teams build agents for rule-shaped work, which costs more and proves less; the PM's job is to stop the count from becoming the strategy.

The follow-up: "Which one ships first?" → the provable one: high volume, low damage per mistake, and an existing human process to compare against, not the flagship.

Red flag: agreeing and listing features, or refusing without an alternative.

• Frame the user and pain with numbers: which businesses, how many emails, minutes per reply, what a slow or wrong reply costs them.
• Slice the work: order status is a lookup (a tool, not generation); returns need policy; complaints need tone and escalation; sales questions need product facts.
• Autonomy per slice: auto-send order status once proven; draft-only for complaints, where the owner sends; never promise refunds the policy does not allow. That is a tool with a cap.
• Measure what users generate: share of drafts sent without edits, edit distance, time to reply, paired with a sampled quality audit, because lazy acceptance inflates "sent unedited".
• Launch: shadow drafts first, then per-slice widening.

The follow-up: "Users say the replies sound robotic." → a tone slice with human ratings; give the model the business's own past replies as style examples; keep facts coming from tools, not from the style.

Red flag: starting from the model or the interface; one accuracy target for everything.

• Per action, never per product: showing options, drafting, sending, paying and deleting are different decisions.
• Five levels: acts alone, acts and is monitored, acts inside a veto window, needs a named approver every time, or is not delegated. Irreversible actions with a large cost of error stay with a person.
• A human hold lowers the bar: it cuts the damage of a mistake to the reviewer's time, which is how a feature ships safely before it could meet an unaided bar.
• The insight: human review fails quietly through automation bias. If reviewers approve 99% of actions in under five seconds, it is not oversight; measure approval rates and times, and seed known-bad cases.

The follow-up: "Reviews are slowing users down." → remove review from reversible actions and add a veto window, rather than raising autonomy on the irreversible ones.

Red flag: "always keep a human in the loop" or "automate everything", with no per-action reasoning.

• Right: offline golden sets from real inputs, labelled by people, per slice, reported with lower bounds.
• Useful: online outcomes (tasks completed, time saved, the feature's retention) against a baseline.
• Cost: cost per task, review time added, retries.
• Safe: policy violations, harmful outputs, escalations.
• The insight: LLM features have metric traps. Thumbs-up rates are self-selected; engagement can rise when quality falls, because users regenerate. Treat regenerations and immediate edits as negative signals.

The follow-up: "Thumbs-up rose ten points after a model change. Ship?" → did the vote rate change, which slices moved, did regenerations and cost change, and what does an A/B on the outcome metric say?

Red flag: one metric, or "accuracy" with no named set.

• What set, built how: real traffic or curated, labelled by whom, and whether it includes the cases the system fails and the ones it should decline.
• How many cases: 92% on 200 has a lower bound of 88.2%, so a 90% bar is not yet proven.
• Where the 8% sits: a risky slice of 30 cases at 90% has a Wilson lower bound near 74%.
• Against what bar: derived from what a mistake costs against what a success saves, per slice, and against the human baseline, which is never 100%.
• The plan: launch the proven slices, keep the others assisted, and collect evidence in shadow.

The follow-up: "The evaluation set was generated by the model." → it is a mirror, not evidence; rebuild it from real inputs with human labels.

Red flag: "92% is great", or "it depends" with nothing after it.

• Start from the slices and bars, not the technology. Most product value comes from context (data, tools, workflow) rather than from weights.
• API first, behind an interface: the fastest learning and a two-way door.
• Fine-tune when a measured gap on a slice survives prompting and retrieval and the volume pays for the data and upkeep; open weights when residency, latency or unit cost at scale demand it.
• Count three-year cost, people included.
• The insight: the moat is rarely the model. It is proprietary data loops, integration into the workflow, trust, and the evaluation set, which is an asset competitors cannot copy.

The follow-up: "A competitor fine-tuned and claims ten points better." → on which benchmark, against your slices? A public benchmark is weaker evidence than your own golden set.

Red flag: "fine-tuning is always better", or a technology preference with no cost.

• Diagnose before fixing: was the right passage retrieved? If yes, was it cited? If cited, does it actually support the claim?
• Each answer has its own fix: retrieval (chunking, hybrid search, reranking), citation (a contract test that every factual claim carries one), verification (an entailment check, or abstaining when support is missing).
• For money, stop generating policy: look it up with a tool and quote it verbatim.
• Measure the rate: a sample of cited answers checked for support, every release.
• The insight: a PM who can separate retrieval from generation failure prioritises correctly; "use a better model" rarely fixes retrieval.

The follow-up: "How often does this happen?" → you do not know until you sample; propose the check.

Red flag: "the model hallucinated; we'll improve the prompt."

• Diagnose from the per-call log: tokens per call, share on the expensive model, cache hit ratio, attempts per case. Confirm the four ratios multiply to the gap.
• Fix in order of (factor − 1) ÷ days to fix: routing easy calls to a cheaper model and trimming context are often fast; add a breaker on loops and tokens per case.
• Report two numbers with a trajectory, before finance asks again.
• The insight: a bill that multiplies on flat traffic is a design outcome; each factor traces to a decision somebody can revisit.

The follow-up: "Finance wants a hard cap today." → a per-case breaker, yes; a global freeze stops the learning and teaches nothing about which signature caused it.

Red flag: "negotiate with the vendor" first, or cutting the feature before diagnosing it.

• Situation and task: what was promised, and the decision that was yours.
• Action: the evidence you gathered (the bar, the lower bound, the cost per case, the shadow run).
• Result, with the number: what shipped instead and what it saved or avoided.
• The change: the process you altered so the next project cannot repeat it (an AI-fit record now required, a lower bound now in every report).
• The insight: interviewers are testing whether you can disappoint a stakeholder with data and keep the relationship.

The follow-up: "What did you get wrong?" → own one specific misjudgement, not a humble-brag.

Red flag: no numbers, blaming the model or engineering, or no change afterwards.

• Classify each request: configuration (the product can already expose it), service (bespoke work an FDE delivers), or product (a capability many customers need).
• Look for the pattern underneath: three different asks may be one need, such as an approval step before money moves.
• Weigh the evidence: customers affected, revenue at stake, effort, and fit with the product's model.
• Defend the call both ways: to the customer, what they get and when; to product, the evidence.
• The insight: the FDPM's real product is the boundary between product and services. Saying yes to everything turns a product company into a consultancy one fork at a time.

The follow-up: "The largest customer will churn without their feature." → a time-boxed service with an owner and a sunset, or a product commitment if it matches the pattern, never an unowned fork.

Red flag: "whoever pays the most."