Home / Guides / Guide

AI Engineer Take-Home Assignment: A Worked Example With Rubric

By Saurav Sharma · November 18, 2026

This page is an original practice take-home for AI engineer roles. It is a worked example with a full brief, a small document set, eight test questions, a scoring rubric, and a two-evening plan. You practice three skills that job postings mention often: finding the right help article (retrieval), testing answers against an answer key (evals), and saying so when the articles do not cover a question.

Why practice this task

Job postings tell you what the work looks like. From aijobprep's study of 3,647 AI-engineering job postings at 198 companies, collected August 2026, evals or benchmarking appear at 135 companies (693 postings). RAG, short for retrieval-augmented generation, appears at 80 companies (364 postings). Vector databases or embeddings appear at 87 companies (359 postings). Large language models (LLMs) appear at 165 companies. Observability and monitoring appear at 154.

In short, more companies mention evals than RAG. A support assistant that finds passages, cites its sources, and comes with a test script covers all of these in one small project. We also searched the postings for the word "hallucination" and found it in 35 of them. We counted a posting if its description matched that word, ignoring case.

Check: open three postings for roles you want. Underline every phrase about evals, retrieval, or reliability. Those phrases tell you which parts of this task to spend extra time on.

Keep in mind that across the same 3,647 postings, only 5 mention a take-home assignment. Most companies do not say in the posting what their interview format is. This page does not claim any company uses this exact task. Use it to practice the skills, whatever format your interview takes.

The brief, as a candidate would receive it

Here is the assignment text. Read it as if it arrived by email.

Build and evaluate a support assistant. You have six short help-center articles. Their facts are in the table below; save each row as a text file before you start. Build a command-line tool that takes a customer question and returns an answer based only on those articles. Time limit: 4 hours of work. Stop at 4 hours and write down what you would do next.

Deliverables: (1) the code, (2) an answer for each of the 8 test questions, with citations to the article each answer came from, (3) an eval script that prints pass or fail for each question, (4) a README with your design choices, tradeoffs, and a cost estimate per 1,000 questions.

Rules: use any language and any model API you have access to. You may use an AI assistant to write code. Be ready to explain every line.

Check: set a 4-hour timer before you start. The time box is part of the test. A reviewer wants to see what you cut and why.

The sample document set

These six fictional articles belong to an invented product called Brightdesk, a scheduling tool. Create one text file per row, using the filename shown. Use only the listed facts, so your answers match the answer key. A few sentences per file is enough.

FileTitleKey facts to include
a1.txtReset your passwordReset link expires after 30 minutes. Reset needs the account email. Accounts with single sign-on reset through the identity provider.
a2.txtPlans and billingTwo plans: Basic at 10 dollars per seat per month and Team at 18 dollars. Billing is monthly. Invoices go to the account owner.
a3.txtCancel your subscriptionOnly the account owner can cancel. Cancel in Settings, then Billing. Access continues to the end of the paid month. No refunds for partial months.
a4.txtCalendar syncSyncs with Google Calendar and Outlook. Sync runs every 15 minutes. Apple Calendar is not supported.
a5.txtExport your dataAccount owners can export bookings as CSV from Settings, then Data. Exports are ready within 10 minutes.
a6.txtTeam rolesThree roles: Owner, Admin, Member. Admins can invite members. Only Owners can change billing or delete the account.

The numbers above are invented for this exercise. Keep them in your files so the test questions have known right answers.

The eight test questions

The documents answer five of the test questions. They do not answer the other three.

#QuestionExpected result
1How long is the password reset link valid?30 minutes, cite a1
2How much does the Team plan cost per seat?18 dollars per month, cite a2
3I am an admin. Can I invite a member and cancel the subscription?You can invite members (a6). Only the account owner can cancel (a3).
4Does Brightdesk work with Apple Calendar?No, not supported, cite a4
5How do I get a CSV of my bookings?Account owners export bookings as CSV from Settings, then Data, cite a5
6Do you offer a student discount?Say the help center does not cover this. No sources.
7Can I pay with a purchase order?Say the help center does not cover this. No sources.
8Is there a mobile app for Android?Say the help center does not cover this. No sources.

Questions 1 to 5 have answers in the files. Questions 6, 7, and 8 do not, so the right behavior is a refusal. Question 3 needs two articles. Question 4 has a negative answer ("No, not supported"). The assistant should give that answer with its source instead of saying it cannot help.

Check: write the expected answer for each question before you write any code. This is your answer key. Without it you cannot grade yourself.

What a strong submission contains

1. Retrieval

With six short articles, you could paste them all into the prompt. That is a valid baseline and you should say so. Then build real retrieval anyway, because reviewers want to see that you can. Split each article into short passages (chunks), rank them by keyword overlap or by embeddings (numeric vectors that capture meaning), and return the top three. Print which chunks were retrieved for each question.

2. Answers with citations

Each answer names the article it came from, such as "(source: a3)". Question 3 needs two sources, for example {"answer": "You can invite members. Only the account owner can cancel.", "sources": ["a6", "a3"]}. Ask the model to return JSON with an answer field and a list of source IDs, then check in code that every cited ID was in the retrieved set.

3. Refusing when the documents do not answer

This is where many submissions fail. Two layers work well. First, if the best retrieval score is low, return "I could not find this in the help center" without calling the model. Second, tell the model it may answer only from the provided text and must say so when the text lacks the answer. Test both layers separately.

4. An eval script

Store the eight questions in a file with the expected behavior: answer or refuse, plus a fact that must appear, such as "30". The script runs all eight and prints PASS or FAIL for each, plus a total. For questions 1 to 5, check the full fact ("30 minutes", not just "30") and compare the source IDs with the answer key. For questions 6 to 8, check that the assistant returns your not-covered message with an empty source list.

5. A README with tradeoffs and cost

Cover these points in under one page: what you chose and why, what you cut because of the time box, where the system still fails, and cost per 1,000 questions. Compute cost from the token counts your script logs and the price on your model provider's pricing page. Write the price and the date you read it.

Check: run your eval script twice. If results change between runs, say so in the README and explain what you would do about it, such as setting the model's temperature to 0 to reduce randomness, or running each question several times.

Scoring rubric

Use this to grade your own work. A total of 100 points is available.

CriterionWeakStrongPoints
RetrievalPastes everything into the prompt, no retrieval output shownChunking, ranking, retrieved chunks printed per question15
Answer correctnessGets 2 or fewer of questions 1 to 5 rightGets questions 1 to 5 right, including question 3, which needs two sources20
CitationsMissing, or cites articles that were not retrievedEvery answer cites the right article, and code verifies the IDs15
RefusalsInvents an answer to the student discount questionRefuses all three unanswerable questions, still answers question 4 (a "no" with a citation)20
Eval scriptManual reading of outputsAutomated pass or fail per question, runs in one command15
READMEDescribes what the code doesNames tradeoffs, known failures, and a cost per 1,000 questions with the price source10
Code qualityOne 300-line file, keys in codeSmall functions, keys from environment, clear run instructions5

Check: find the row where you lost the most points, fix one missed requirement there, and score that row again.

Common mistakes

A two-evening practice plan

Evening 1: build (about 2 hours)

  1. Write the six articles and the answer key (30 minutes).
  2. Build chunking and retrieval, and print the top three chunks for each question (45 minutes).
  3. Add the answer step with JSON output and citations (45 minutes).

Evening 2: measure and explain (about 2 hours)

  1. Write the eval script and get a first pass or fail table (45 minutes).
  2. Fix the two worst failures. Rerun and record before and after scores (45 minutes).
  3. Write the README with tradeoffs and cost, then explain the project aloud in 5 minutes (30 minutes).

Check: the final spoken explanation is practice for the follow-up conversation, where you defend your choices. If you cannot explain why you picked your refusal threshold, change it until you can.

Next, practice the questions that follow a take-home like this one. See AI engineer interview questions, and for customer-facing roles see forward-deployed engineer questions and ML engineer questions.