GenAI engineer interview questions: RAG, evals and cost

• Build a labelled set with gold passages, then measure the two halves separately: retrieval recall at k (is the needed passage in the top k?) and, for cases where it was, faithfulness (does the answer follow from the passage?).
• Bucket the failures, because the fixes differ: chunking by meaning rather than by size, hybrid keyword-plus-vector search, a reranker, query rewriting, or, for generation, citations per claim and abstaining when support is missing.
• The insight: raising k often lowers accuracy. Models use information in the middle of a long context less reliably than at the edges, so extra passages dilute the right one. Measure accuracy against k; it peaks and then falls.

The follow-up: "Recall is 95% and answers are still wrong." → a generation failure: check faithfulness, position of the passage, conflicting passages, and whether the prompt allows the model to decline.

Red flag: "use a bigger model" before measuring either half.

• Before launch: a golden set from real inputs, labelled by people, sliced by kind of case, including cases the system fails and cases it should decline; a checker per kind of step; a CI gate per slice reported with lower bounds.
• Model judges, calibrated: a judge from a different model family, a rubric, and a measured agreement rate with human labels. Judges are known to favour the first answer shown, longer answers, and their own model family's outputs: randomise order and control for length.
• After launch: a shadow run against the current process, then a canary, then a sampled human review and a watch on the mix of outputs for drift.
• The insight: a golden set built from cases the system already passes is a mirror; it can only go down.

The follow-up: "How big should the set be?" → sized per slice by how close the score is to its bar: the cases needed grow with the square of the gap.

Red flag: public benchmarks as the main evidence.

• Retrieval for knowledge that changes or must be cited.
• Prompting and examples for behaviour you can describe.
• Fine-tuning for consistency at scale (format, style, a narrow task) or to distil a large model's behaviour into a smaller, cheaper, faster one, once you have enough good examples and a gap that prompting did not close.
• Not for facts: fine-tuned knowledge goes stale silently and cannot be cited.
• The insight: fine-tuning moves cost from inference to data and operations: every base-model update means re-training and re-evaluating. Measure the gap it closes before paying that tax.

The follow-up: "The fine-tuned model is better on the benchmark and worse in production." → the benchmark is not your distribution; evaluate on your own slices.

Red flag: "fine-tune it on our documents so it knows our policies."

• Measure the split first. Model time: cap output tokens, stream, use a faster model on easy routes, cache a long stable prefix. Tool time: parallelise independent calls, cache, set timeouts. Orchestration: fewer turns; route known paths to a workflow.
• Ask which clock the requirement is about: time to first token or total time. Streaming the first token in 400 ms can make an eight-second answer feel fast.
• The insight: most latency programmes optimise the model clock because it is visible, while the orchestration clock (turns multiplied by round trips) is the one that dominates agentic systems.

The follow-up: "Streaming is not allowed; the output is JSON." → then fewer turns, smaller outputs, parallel tools, and a faster model per slice where its bar still holds.

Red flag: "use a faster model" without a measurement.

• Calls per task first: it multiplies the instruction, schema, history and retrieval tokens on every turn.
• Then the taxes: retrieval (top-k set by habit), schema (tool definitions sent on every call), history (unbounded buffers), reasoning tokens after a prompt change.
• Then the four signatures: tokens per call, share on the expensive tier, cache hit ratio, attempts per case.
• The insight: a timestamp, request ID or user name near the top of a system prompt silently destroys prompt caching, because caches match an exact prefix. Put stable content first and variables last.

The follow-up: "Caching is on and the hit rate is 3%." → find what varies in the prefix, and check the provider's minimum cacheable length and cache lifetime.

Red flag: "negotiate a discount."

• Define slices and bars first, then test two or three candidates on the golden set.
• Route by slice: the cheapest model that clears each slice's bar, a stronger one for the rest.
• Pin versions, log which model answered every response, and plan the fallback: a fallback to a larger model is a cost cliff, and a fallback to a smaller one is a silent quality drop.
• Re-evaluate on every model update, as a change like any other.
• The insight: routing by difficulty usually saves more than any single model switch, and logging the answering model is the only way to detect silent failover.

The follow-up: "A new model tops the leaderboard." → run it through the harness per slice; adopt it where it clears the bar more cheaply.

Red flag: "the best model on the leaderboard."

• Treat every retrieved document and tool output as untrusted data, and mark it as such in the prompt.
• Limit what a tricked model can do: fewer tools while untrusted content is in context, least privilege per tool, caps in signatures, confirmation tokens the model cannot create for consequential actions, egress allowlists.
• Filter outputs for data exfiltration patterns, and log every tool call.
• Test continuously: an attack suite on every prompt, tool or context change, from every entry point.
• The insight: the question is not whether the model can be tricked (it can) but what is the worst it can do when it is. Injection is a permissions problem more than a prompt problem.

The follow-up: "A partner API adds a free-text field." → a new entry point: extend the attack suite to it before it reaches production.

Red flag: "tell the model to ignore instructions in documents."

• Constrain: use native structured output or tool-use schemas, small and well described.
• Validate with a schema parser, then validate the meaning with exact checks: dates in range, IDs that exist, totals that add up.
• Repair with a bounded retry that feeds back the validation error.
• Monitor the parse-failure and repair rates per slice.
• The insight: well-formed is not correct. A perfectly valid JSON object with the wrong account number is the dangerous failure, and only a semantic check catches it.

The follow-up: "Repairs succeed 99% of the time." → and the 1% goes where? Route it to a person, and count repairs as a cost.

Red flag: regular expressions over free text.

• Short-term: the current conversation, with a summary once it passes a threshold.
• Long-term: extracted facts and preferences with their provenance, a retention period, and user controls to view and delete them; retrieved by relevance, not replayed wholesale.
• Never: raw personal data, payment details, full records. Nothing stored is the best leak control.
• The insight: memory can make an assistant worse. A wrong or stale memory is a persistent hallucination that the user cannot see; show what is remembered and let people correct it.

The follow-up: "Extraction runs asynchronously and lags." → use short-term memory for the current session and design for the gap.

Red flag: "store everything in a vector database."

• The numbers: cost per task, latency split, golden-set size and slices, lower bounds.
• The failure: how it was found, which signal, and the root cause.
• The change: the guard, test or check that now makes that failure impossible or visible.
• The insight: interviewers use this question to check everything above. A candidate who shipped something real can answer it with numbers without pausing.

The follow-up: "What would you build differently?" → a specific design decision, and why.

Red flag: no numbers, or a failure that was "the model's fault".