Home / Guides / Guide

LLM System Design Interview: A Framework and Worked Example

By Saurav Sharma · November 18, 2026

An LLM (large language model) system design round gives you a vague product request and 45 minutes to design it out loud. The interviewer wants to see an order of thinking: who uses it, what data it touches, how you know it works, what it costs, and what happens when it breaks. This guide gives a six-step framework for that order, one fully worked example, five follow-ups that force a redesign, and five prompts to practice.

Which topics AI engineering job postings mention

These numbers come from aijobprep's study of 3,647 AI-engineering job postings at 198 companies, collected August 2026. A company counts once for a topic if at least one of its postings mentions it. The counts come from aijobprep's August 2026 study of public job postings.

TopicCompaniesPostings
LLMs1651,486
Observability and monitoring1541,038
Evals and benchmarking135693
Distributed systems123698
Inference optimization113541
Vector databases and embeddings87359
RAG (retrieval-augmented generation)80364

Two points matter for your preparation. First, observability (monitoring a system in production) and evals (tests of answer quality) both appear at more companies than RAG does. A design answer that stops at "embed the documents and search them" covers the narrowest of these topics. Second, distributed systems appears at 123 companies, so practice explaining what happens when requests arrive faster than the system can process them, and when a part fails. RAG still matters: it is the standard answer to "answer questions over private documents", and the worked example below uses it.

Check: open the job posting for your target role and mark which of these seven topics it names. Start with the topics your target posting names.

The six-step framework

Say the step names out loud as you go. The interviewer can then follow you and steer you.

  1. Clarify users and the success metric. Ask who asks questions, what a good answer is, and what a wrong answer costs. Pick one number to improve, such as the share of questions answered correctly with a valid citation.
  2. Data and access permissions. Ask where the data lives, how often it changes, how large it is, and who may see which part.
  3. Retrieval and model choice. Choose how you find the right text (keyword search, vector search, or both) and which model writes the answer. Say why you chose each.
  4. Evaluation. Say how you will measure quality before launch and after. Name the test set, the metrics, and who writes the correct answers for it.
  5. Cost, latency, and fallbacks. Estimate cost per request. Set a latency budget. Say what the user sees when a part fails.
  6. Monitoring and rollout. Say what you log, which alerts you set, and how you ship changes without hurting users.

Check: write the six names on a sticky note. During a practice answer, tick each one off. Before you finish, make sure you have covered step 4 (testing answers) and step 5 (cost and failure handling).

Worked example: search and question answering over a company's internal wiki

The prompt: "Design search and question answering over our internal wiki. Employees type a question and get an answer with sources."

Step 1: users and success metric

Ask three questions. Who uses it (all employees, or one team)? What do they do today (keyword search in the wiki tool)? What is the cost of a wrong answer (a wasted hour, or a wrong policy applied)? Then state your metric: the share of questions where the answer is correct and every claim points to a source the user may open. Add a safety metric that must stay at zero: the share of answers that show text from a document the user should not see. That one must be zero.

Step 2: data, updates, and permissions

The wiki holds pages, attachments, and comments. Each page has an owner, a space, and an access list. Describe the ingestion path as an ordered list:

  1. A connector reads the wiki's change feed (new, edited, and deleted pages) and writes events to a queue.
  2. A worker fetches each changed page and its access list.
  3. The worker splits the page into chunks along headings, so each chunk keeps its section title.
  4. The worker creates an embedding (a list of numbers that represents the meaning of the text) for each chunk.
  5. The worker writes the chunk, its embedding, the page ID, the page version, and the access list to the index.
  6. For a deleted page, the worker removes all chunks with that page ID.

Updates arrive through the change feed, so freshness is the delay between an edit and step 5. Say the target out loud and say how you measure it: store the edit time and the index time, and chart the gap.

Permissions need the most care. Store each page's access list next to its chunks. At query time, look up the asking user's groups and filter the search to chunks whose access list includes one of those groups. Filter inside the search query. Apply the access check during search, so only passages the employee may read ever reach the model. Filtering afterward also leaves too few results when most matches are restricted. Also sync permission changes on their own schedule, since a page can change access without any edit to its text.

Step 3: retrieval and model choice

The query path, as an ordered list:

  1. The API authenticates the user and loads their groups.
  2. The API rewrites the question if needed (for example, it expands "PTO" to "paid time off").
  3. The search runs two ways at once: keyword search for exact names and codes, and vector search for meaning. The permission filter applies to both.
  4. A reranker (a small model that reorders the top results by relevance) keeps the best few chunks.
  5. The prompt builder places the chunks in the prompt with numbered labels and tells the model to answer only from them and to cite the labels.
  6. The model returns the answer.
  7. A check confirms that each citation points to a retrieved passage, and the API returns the answer with links to the source pages. Your test set should also check that the cited passage supports the claim.
  8. If the retrieved chunks do not contain the answer, the system says so and shows the closest pages.

On model choice, say what you would test. A mid-size model usually writes a good grounded answer from a few chunks. A larger model costs more per question and may add little here. Give both models the same questions and passages from your test set (step 4), compare how often each answer is correct and supported by its sources, then compare cost per question.

Step 4: evaluation

Build a test set of real questions. Collect them from the existing search logs and from a few team leads. For each question, record the right source page and a short reference answer. Then score three things separately:

Add a set of questions the wiki cannot answer. The right behavior there is "I could not find this". Run the test set on every change to the prompt, the chunking, or the model. After launch, add thumbs up and down buttons and review the thumbs-down cases weekly.

Step 5: cost, latency, and fallbacks

Cost per question is the sum of four parts: query embedding, search, reranking, and the model call. The model call dominates, so use this formula: (prompt tokens x input price) plus (answer tokens x output price). Prompt tokens count the question, the instructions, and the retrieved chunks. Fewer, better chunks cut cost and often improve quality. Say you would read the current price from the model vendor's price page and put the result in a spreadsheet with a column for each part. Cache answers for repeated questions, but reuse a cached answer only if the employee can read every source page it used and those pages have not changed.

For latency, stream the answer so the user sees words early, and give each stage a time budget.

For fallbacks, walk through the failures in order. If the model is down, retry once, then switch to a second model or provider. If both fail, show the search results alone, ranked, with the matching text highlighted. The user still gets value. If the reranker fails, skip it. If the vector index is down, use keyword search only.

Step 6: monitoring and rollout

Log the question, the chunk IDs retrieved, the answer, the citations, the latency of each stage, and the token counts. Redact or restrict logs, since questions can contain sensitive text. Alert on error rate, p95 latency (the time under which 95 percent of requests finish), cost per question, the share of "could not find" answers, and the index freshness gap. Roll out to one team first. Compare the old and new versions on the same test questions and read that team's negative feedback before widening access. Keep the previous prompt and model version ready for an immediate rollback.

Check: say this design out loud in 10 minutes, timed. If step 4 or 5 took less than a minute each, slow down there.

Five follow-ups that force a redesign

Interviewers use follow-ups to test whether your design bends. For each one below, name the part that breaks first, then change it.

1. "We now have 10x more documents."

Name what grows: index size, ingestion time, and search latency. Strong answer: first measure whether search has slowed. If it has, split (shard) the index by wiki space or team, use an approximate nearest-neighbor index (a faster vector search that can miss a few matches, so measure recall), and keep the keyword index alongside it. Add ingestion workers when the queue of changed pages grows. Cut stored data by removing duplicate and stale pages. Re-run retrieval tests, because more documents mean more near-misses and the right page may fall below the cutoff. Add a first-stage filter by space or recency to shrink the candidate set.

2. "One team's documents must never reach another team."

Treat this as a hard boundary. Strong answer: enforce the filter inside the index query with the user's groups, and for the most sensitive teams use a separate index per team so a bug in a filter cannot cross the line. Do not share a cache across permission groups. Restrict log access by team. Add a test that signs in as a user in team A, asks questions whose answers exist only in team B's pages, and fails the build if any chunk or citation from team B appears. Run that test nightly.

3. "Answers must come back in under 2 seconds."

Split the budget across stages and measure each. Strong answer: run keyword and vector search in parallel, use a smaller reranker or drop it, retrieve fewer chunks to shorten the prompt, pick a faster model, and stream the answer so the first words arrive early. Cache frequent questions. Trade a measured amount of quality for speed and say how you would verify the loss on the test set. First ask the interviewer whether 2 seconds means the first words appear (time to first token) or the full answer is done.

4. "The eval score dropped after a model upgrade."

Do not push the new model to everyone until the test score recovers. Strong answer: roll back or pin the old version first. Then compare per-question results between versions to find the pattern. Check retrieval scores to confirm the retrieval side did not change. If the new model writes longer answers, formats citations differently, or refuses more often, the grader may be reacting to style. Have a person read a few answers whose scores fell. If the answers got worse, adjust the prompt or keep the old model. If correct answers lost points only for format, fix the scoring rule. Re-run the test set and release only when the score recovers. Make the eval run a required step for any model change from now on.

5. "Users report a wrong answer that cites a real page."

Pull the log for that request. Check whether the right chunk was retrieved, whether the cited chunk supports the claim, and whether the page itself is out of date. Strong answer: add the case to the test set, fix how the system splits, ranks, or uses passages, or ask the page owner to correct outdated content, add a "last updated" date next to each citation, and let users flag pages as stale.

Five more practice prompts

Run the six steps on each. The line after each prompt names the main tradeoff to defend.

  1. A customer support assistant that answers from help articles and past tickets. Tradeoff: answer automatically for speed, or hand off to a person for safety, and where to draw that line.
  2. A code review assistant that comments on pull requests. Tradeoff: catching more issues against flooding authors with low-value comments.
  3. A meeting summarizer for recorded calls. Tradeoff: cost of long transcripts against the risk of missing action items.
  4. A document extraction pipeline that turns invoices into structured data. Tradeoff: a cheap model with human review for low-confidence fields against a costly model for everything.
  5. An agent that books travel using company tools. Tradeoff: autonomy against approval steps, and how you make tool calls safe to retry.

Check: pick two prompts. For each, write the success metric and the three fallbacks in five lines. Then explain them to someone out loud.

Practice plan for one week

Practice the questions behind these rounds on the AI engineer question page. For adjacent roles, see ML engineer, MLOps engineer, and cloud solutions architect.