Machine learning and generative AI system design interview guide

What an AI system design interview measures

A strong answer is not a diagram made from fashionable components. The interviewer is testing whether you can turn an ambiguous product request into a measurable contract, design every consequential data and control path, and defend decisions as constraints change. In machine learning and generative AI interviews, you must reason about uncertain model behaviour without abandoning ordinary systems engineering.

Your answer should make six things visible: the user and task, the definition of success, the request and data flows, the AI-specific decisions, the failure and trust boundaries, and the evidence you will use to operate the system.

1. Clarify the product contract before choosing technology

Name the user, the job they are trying to complete, and the harm caused by a bad answer. Ask about traffic, data volume, freshness, privacy, geography, latency, cost, and availability. Turn vague goals such as “accurate” or “fast” into observable targets. For an enterprise RAG assistant, useful targets might include citation correctness, permission-safe retrieval, answer coverage, P95 time to first token, cost per resolved question, and a safe fallback rate.

Do not select a model, vector database, or agent framework until the contract tells you what those components must accomplish.

2. Draw every consequential path

Separate the online request path from offline ingestion and evaluation. The request path might include identity, policy, query understanding, retrieval, reranking, context construction, model inference, citation validation, and response streaming. The ingestion path might include connectors, parsing, chunking, metadata, embeddings, indexing, authorization updates, freshness, and deletion.

Also draw state, tool execution, feedback, observability, and failure paths. A component list cannot explain where evidence, authority, or latency moves. Label trust boundaries and identify which system—not the model—owns identity, authorization, budgets, and irreversible actions.

3. Defend the model and context strategy

Compare model candidates using task quality, latency, price, context limits, structured-output reliability, deployment constraints, safety, and switching cost. State when a smaller routed model is enough and when a larger model earns its cost. Separate prompt changes, retrieval, fine-tuning, and deterministic code: each solves a different class of problem.

For retrieval-augmented generation, explain source selection, chunk boundaries, metadata filters, hybrid lexical and vector retrieval, reranking, context packing, citations, freshness, and deletion. Retrieval adds evidence; it does not guarantee truth. Your evaluation must test whether the correct evidence was found and whether the answer used it faithfully.

4. Bound agents and tool use

Use a deterministic workflow where the sequence is known. Introduce model-directed planning only when the task genuinely requires judgment over changing states. Define the agent loop explicitly: observe, choose an action, validate it, execute within permissions, record the result, and decide whether to continue.

Every tool needs a typed contract, authentication, authorization, idempotency where possible, timeouts, bounded retries, and an audit trail. Put approvals before consequential actions. Set step, time, and cost budgets plus stop conditions so the system cannot loop forever or silently expand its authority.

5. Evaluate the task, not the prose

Build a representative case set from real task categories, rare but costly failures, adversarial inputs, permission boundaries, and changed data. Score dimensions that match the product contract: task completion, retrieval recall, citation support, structured-output validity, unsafe action rate, latency, and cost. Keep deterministic checks where they work; use human review for nuanced judgment; treat LLM judges as fallible instruments that must be calibrated.

Describe pre-release gates, online monitoring, sampled review, regression thresholds, and rollback. A single offline average hides important slices, so report performance by user group, content type, language, risk level, and difficulty.

6. Design reliability, security, cost, and latency together

Budget latency across retrieval, model inference, tools, and streaming. Budget cost per request and per successful task. Use caching only where identity, freshness, and privacy permit it. Explain timeouts, circuit breakers, backpressure, queues, fallbacks, provider failure, and graceful degradation.

Threat-model prompt injection, poisoned documents, data exfiltration, insecure tool calls, cross-tenant retrieval, excessive agency, and sensitive logging. The model never grants access: authorization must be enforced before retrieval, before tool execution, and before returning citations or data.

A practical 45-minute interview structure

1. 0–5 minutes: clarify users, task, risk, scale, constraints, and success metrics.
2. 5–10 minutes: state assumptions and sketch the high-level request and ingestion paths.
3. 10–25 minutes: deepen the model, data, retrieval, workflow, state, and trust decisions.
4. 25–35 minutes: cover evaluation, reliability, security, observability, latency, and cost.
5. 35–45 minutes: answer follow-ups, compare alternatives, find bottlenecks, and summarise trade-offs.

Narrate why each choice follows from the product contract. When the interviewer changes a constraint, update the affected boundary instead of redrawing the entire system.

Questions your design should survive

• What changes if traffic grows by 100×?
• How do you delete a document and every derived representation?
• How do you prove one tenant cannot retrieve another tenant’s data?
• What happens when the model times out after a tool has already executed?
• Which metric would detect a retrieval regression before users report it?
• How do you reduce cost without hiding a quality loss?
• Which part can degrade safely when a provider is unavailable?