AI Engineer Take-Home Assignment: A Worked Example With Rubric
By Saurav Sharma · November 18, 2026
This page is an original practice take-home for AI engineer roles. It is a worked example with a full brief, a small document set, eight test questions, a scoring rubric, and a two-evening plan. You practice three skills that job postings mention often: finding the right help article (retrieval), testing answers against an answer key (evals), and saying so when the articles do not cover a question.
Why practice this task
Job postings tell you what the work looks like. From aijobprep's study of 3,647 AI-engineering job postings at 198 companies, collected August 2026, evals or benchmarking appear at 135 companies (693 postings). RAG, short for retrieval-augmented generation, appears at 80 companies (364 postings). Vector databases or embeddings appear at 87 companies (359 postings). Large language models (LLMs) appear at 165 companies. Observability and monitoring appear at 154.
In short, more companies mention evals than RAG. A support assistant that finds passages, cites its sources, and comes with a test script covers all of these in one small project. We also searched the postings for the word "hallucination" and found it in 35 of them. We counted a posting if its description matched that word, ignoring case.
Check: open three postings for roles you want. Underline every phrase about evals, retrieval, or reliability. Those phrases tell you which parts of this task to spend extra time on.
Keep in mind that across the same 3,647 postings, only 5 mention a take-home assignment. Most companies do not say in the posting what their interview format is. This page does not claim any company uses this exact task. Use it to practice the skills, whatever format your interview takes.
The brief, as a candidate would receive it
Here is the assignment text. Read it as if it arrived by email.
Build and evaluate a support assistant. You have six short help-center articles. Their facts are in the table below; save each row as a text file before you start. Build a command-line tool that takes a customer question and returns an answer based only on those articles. Time limit: 4 hours of work. Stop at 4 hours and write down what you would do next.
Deliverables: (1) the code, (2) an answer for each of the 8 test questions, with citations to the article each answer came from, (3) an eval script that prints pass or fail for each question, (4) a README with your design choices, tradeoffs, and a cost estimate per 1,000 questions.
Rules: use any language and any model API you have access to. You may use an AI assistant to write code. Be ready to explain every line.
Check: set a 4-hour timer before you start. The time box is part of the test. A reviewer wants to see what you cut and why.
The sample document set
These six fictional articles belong to an invented product called Brightdesk, a scheduling tool. Create one text file per row, using the filename shown. Use only the listed facts, so your answers match the answer key. A few sentences per file is enough.
| File | Title | Key facts to include |
|---|---|---|
| a1.txt | Reset your password | Reset link expires after 30 minutes. Reset needs the account email. Accounts with single sign-on reset through the identity provider. |
| a2.txt | Plans and billing | Two plans: Basic at 10 dollars per seat per month and Team at 18 dollars. Billing is monthly. Invoices go to the account owner. |
| a3.txt | Cancel your subscription | Only the account owner can cancel. Cancel in Settings, then Billing. Access continues to the end of the paid month. No refunds for partial months. |
| a4.txt | Calendar sync | Syncs with Google Calendar and Outlook. Sync runs every 15 minutes. Apple Calendar is not supported. |
| a5.txt | Export your data | Account owners can export bookings as CSV from Settings, then Data. Exports are ready within 10 minutes. |
| a6.txt | Team roles | Three roles: Owner, Admin, Member. Admins can invite members. Only Owners can change billing or delete the account. |
The numbers above are invented for this exercise. Keep them in your files so the test questions have known right answers.
The eight test questions
The documents answer five of the test questions. They do not answer the other three.
| # | Question | Expected result |
|---|---|---|
| 1 | How long is the password reset link valid? | 30 minutes, cite a1 |
| 2 | How much does the Team plan cost per seat? | 18 dollars per month, cite a2 |
| 3 | I am an admin. Can I invite a member and cancel the subscription? | You can invite members (a6). Only the account owner can cancel (a3). |
| 4 | Does Brightdesk work with Apple Calendar? | No, not supported, cite a4 |
| 5 | How do I get a CSV of my bookings? | Account owners export bookings as CSV from Settings, then Data, cite a5 |
| 6 | Do you offer a student discount? | Say the help center does not cover this. No sources. |
| 7 | Can I pay with a purchase order? | Say the help center does not cover this. No sources. |
| 8 | Is there a mobile app for Android? | Say the help center does not cover this. No sources. |
Questions 1 to 5 have answers in the files. Questions 6, 7, and 8 do not, so the right behavior is a refusal. Question 3 needs two articles. Question 4 has a negative answer ("No, not supported"). The assistant should give that answer with its source instead of saying it cannot help.
Check: write the expected answer for each question before you write any code. This is your answer key. Without it you cannot grade yourself.
What a strong submission contains
1. Retrieval
With six short articles, you could paste them all into the prompt. That is a valid baseline and you should say so. Then build real retrieval anyway, because reviewers want to see that you can. Split each article into short passages (chunks), rank them by keyword overlap or by embeddings (numeric vectors that capture meaning), and return the top three. Print which chunks were retrieved for each question.
2. Answers with citations
Each answer names the article it came from, such as "(source: a3)". Question 3 needs two sources, for example {"answer": "You can invite members. Only the account owner can cancel.", "sources": ["a6", "a3"]}. Ask the model to return JSON with an answer field and a list of source IDs, then check in code that every cited ID was in the retrieved set.
3. Refusing when the documents do not answer
This is where many submissions fail. Two layers work well. First, if the best retrieval score is low, return "I could not find this in the help center" without calling the model. Second, tell the model it may answer only from the provided text and must say so when the text lacks the answer. Test both layers separately.
4. An eval script
Store the eight questions in a file with the expected behavior: answer or refuse, plus a fact that must appear, such as "30". The script runs all eight and prints PASS or FAIL for each, plus a total. For questions 1 to 5, check the full fact ("30 minutes", not just "30") and compare the source IDs with the answer key. For questions 6 to 8, check that the assistant returns your not-covered message with an empty source list.
5. A README with tradeoffs and cost
Cover these points in under one page: what you chose and why, what you cut because of the time box, where the system still fails, and cost per 1,000 questions. Compute cost from the token counts your script logs and the price on your model provider's pricing page. Write the price and the date you read it.
Check: run your eval script twice. If results change between runs, say so in the README and explain what you would do about it, such as setting the model's temperature to 0 to reduce randomness, or running each question several times.
Scoring rubric
Use this to grade your own work. A total of 100 points is available.
| Criterion | Weak | Strong | Points |
|---|---|---|---|
| Retrieval | Pastes everything into the prompt, no retrieval output shown | Chunking, ranking, retrieved chunks printed per question | 15 |
| Answer correctness | Gets 2 or fewer of questions 1 to 5 right | Gets questions 1 to 5 right, including question 3, which needs two sources | 20 |
| Citations | Missing, or cites articles that were not retrieved | Every answer cites the right article, and code verifies the IDs | 15 |
| Refusals | Invents an answer to the student discount question | Refuses all three unanswerable questions, still answers question 4 (a "no" with a citation) | 20 |
| Eval script | Manual reading of outputs | Automated pass or fail per question, runs in one command | 15 |
| README | Describes what the code does | Names tradeoffs, known failures, and a cost per 1,000 questions with the price source | 10 |
| Code quality | One 300-line file, keys in code | Small functions, keys from environment, clear run instructions | 5 |
Check: find the row where you lost the most points, fix one missed requirement there, and score that row again.
Common mistakes
- Skipping the answer key. You cannot write an eval without known right answers.
- Only testing questions the docs answer. The refusal cases are the real test.
- Refusing too much. Over-strict thresholds make the assistant refuse question 4, where the true answer is "no". Test both directions.
- Trusting model-written citations. Verify in code that each cited ID was actually retrieved.
- Spending most of the time setting up a vector database. Six articles do not need a database. A list in memory is enough. Say so in the README.
- No cost figure. Cost per 1,000 questions is a short calculation. Leaving it out looks like you did not think about production.
- A README that only describes the code. Reviewers can read code. They want your reasoning and your known failures.
A two-evening practice plan
Evening 1: build (about 2 hours)
- Write the six articles and the answer key (30 minutes).
- Build chunking and retrieval, and print the top three chunks for each question (45 minutes).
- Add the answer step with JSON output and citations (45 minutes).
Evening 2: measure and explain (about 2 hours)
- Write the eval script and get a first pass or fail table (45 minutes).
- Fix the two worst failures. Rerun and record before and after scores (45 minutes).
- Write the README with tradeoffs and cost, then explain the project aloud in 5 minutes (30 minutes).
Check: the final spoken explanation is practice for the follow-up conversation, where you defend your choices. If you cannot explain why you picked your refusal threshold, change it until you can.
Next, practice the questions that follow a take-home like this one. See AI engineer interview questions, and for customer-facing roles see forward-deployed engineer questions and ML engineer questions.