AI Engineer Interview Questions (2026)
AI engineer is the title companies use for people who build products on top of LLMs — distinct from ML engineers who train models. Interviews test practical LLM system building: RAG, agents, evals, and cost. These are the questions that show up in real loops.
1.Explain how you'd build a RAG system from scratch. Where do they fail in production?
What a strong answer covers: Cover the pipeline: ingestion, chunking (and why chunk size/overlap matter), embeddings, vector store with metadata filtering, hybrid retrieval (BM25 + dense), reranking, prompt assembly, generation with citations. Then the production failures: bad retrieval (the top cause), stale indexes, context window stuffing that degrades answers, and no eval set. Candidates who lead with failure modes signal real experience.
2.When would you fine-tune instead of using RAG or better prompting?
What a strong answer covers: Fine-tuning teaches behavior and format; RAG provides knowledge. Fine-tune for: consistent output structure at scale, style/tone matching, latency and cost reduction by moving a narrow task to a smaller model, and classification tasks with abundant labels. Don't fine-tune to inject facts that change — that's RAG's job. Mention needing an eval set before any fine-tune, or you can't tell if it worked.
3.How do you evaluate LLM outputs when there's no single right answer?
What a strong answer covers: The question that separates builders from tinkerers. Cover: golden datasets from real usage, rubric-based scoring, LLM-as-judge with known biases (position bias, verbosity bias) and human calibration of the judge, pairwise comparison over absolute scores, and regression tracking in CI so prompt changes can't silently degrade quality. Name a metric you'd track weekly.
4.Design an agent that can answer questions requiring multiple tool calls. What goes wrong?
What a strong answer covers: Cover the loop: model decides tool → executes → observes → repeats, with a termination condition. Failure modes interviewers want to hear: infinite loops, error handling when tools fail, context window growth over long trajectories, compounding hallucination between steps, and cost blowup. Mitigations: max-step limits, structured tool schemas, trajectory evals, and human approval gates on consequential actions.
5.Your LLM app costs 10x what was budgeted. How do you cut costs without gutting quality?
What a strong answer covers: Levers in order of typical impact: prompt caching for repeated context, routing easy queries to smaller models with a classifier, trimming the system prompt and retrieved context, batching where latency allows, and semantic caching of repeated questions. Measure quality on your eval set at each step. Knowing that input tokens usually dominate cost in RAG apps is a strong tell.
6.How do you handle hallucinations in a customer-facing product?
What a strong answer covers: Layered answer: ground responses in retrieved sources with citations, constrain output to what's in context ('say I don't know if the answer isn't present'), post-generation verification for high-stakes claims, confidence thresholds routing to humans, and UX honesty about AI limitations. Zero hallucination isn't achievable; the design question is making errors visible and recoverable.
7.What's your approach to prompt engineering beyond trial and error?
What a strong answer covers: Treat prompts as code: version controlled, tested against an eval set, changed one variable at a time. Techniques worth naming: few-shot examples chosen from real data, structured output schemas, decomposing tasks into chained smaller prompts, and letting the model reason before answering. The discipline (evals gate every change) matters more than any single trick.
8.Compare the tradeoffs of using OpenAI/Anthropic APIs vs self-hosting open-weight models.
What a strong answer covers: APIs: best quality, zero infra, per-token cost, data leaves your boundary (unless via cloud private endpoints), rate limits. Self-hosted: fixed GPU cost that only wins at high sustained volume, full data control, operational burden (vLLM, quantization, GPU availability), and a quality gap that keeps narrowing. The honest answer: most products should start on APIs and revisit at scale or under hard compliance constraints.
9.How would you add memory to a chat application so it remembers users across sessions?
What a strong answer covers: Distinguish memory types: full transcript storage (expensive to reuse), rolling summaries, and extracted structured facts (preferences, entities) stored and retrieved per session. Cover retrieval into the system prompt, update policies (when to write memory), conflict resolution when facts change, and privacy/deletion requirements. Bonus: mention evaluating whether memory actually improves outcomes vs bloating context.
10.A prompt that worked well suddenly performs worse after a model version update. What do you do?
What a strong answer covers: This tests production maturity. Immediate: pin the previous model version if the provider allows, quantify the regression on your eval set. Then: adapt prompts to the new model with eval-gated iteration, maintain a migration checklist for future updates, and never auto-upgrade model versions in production without running evals. Candidates who've lived this have strong specific stories — tell one.
11.What are embeddings, and how would you explain a vector similarity search to a non-technical stakeholder?
What a strong answer covers: Test of both understanding and communication. Technical: dense vectors capturing semantic meaning, cosine similarity, ANN indexes for scale. The stakeholder explanation: 'we turn every document into a point on a map where similar meanings sit near each other; a question drops onto the map and we grab its nearest neighbors.' Practice the plain version — AI engineers translate constantly.
12.How do you stay current when the field changes monthly, and how do you decide what to ignore?
What a strong answer covers: Interviewers want a filter, not a firehose. Strong answers name specific sources and a decision rule: adopt things that survive 2-3 months of scrutiny, test claims on your own evals before believing benchmarks, ignore anything that doesn't change what you'd build this quarter. Mentioning something you evaluated and rejected is more convincing than listing what you follow.
Prepping for a specific job?
Use the prep engine on the homepage — describe your interview and get a tailored question set instantly. Missing a role page? Request it.
Want a human in your corner? 1-on-1 interview prep — $150
A 1-hour session where I use AI to build a prep plan for your exact interview — the role, the company, the round. Strongest for technical interviews (cloud, AI/ML, DevOps) and behavioral rounds: mock questions, answer structuring, and a follow-up question bank tailored to your job description.
Book a session