AI Engineer Interview Questions and Answers

What AI Engineer interviewers assess

This is the real work sitting behind the questions. They are checking whether you have actually done it, not whether you can describe it.

• Retrieval-augmented generation: Chunking, embeddings, hybrid search, re-ranking, and working out whether a bad answer came from retrieval or from generation.
• Evaluation: Building eval sets from real traffic, choosing metrics, using model-graded evals carefully, and running regression checks before every prompt or model change.
• Prompt, tool and agent design: Structured outputs, tool definitions, multi-step agent loops, stopping conditions, and knowing when a fixed workflow beats an autonomous agent.
• Guardrails and safety: Prompt injection, data leakage, permissioned tool use, content filtering, and human approval for actions that cannot be undone.
• Cost and latency: Model routing, caching, streaming, batching, and the trade-off between a large model on every call and a smaller one that is good enough.
• Fine-tuning versus prompting: When better prompts and retrieval are enough, when fine-tuning earns its keep, and what it costs you to maintain a custom model.

Common AI Engineer interview questions with answer guidance

1. How is the job of an AI engineer different from a machine learning engineer?

Keep it practical. ML engineers mostly train, serve and monitor models built from their own data. AI engineers mostly build products on foundation models through APIs or open weights, so the work shifts to retrieval, prompting, tool use, evals and cost control. Say where the two overlap, monitoring and data quality especially, and where you sit on that line yourself. Avoid sounding dismissive of either side.

2. Design a RAG system that answers questions over our internal documentation.

Start with the questions users will ask and how fresh the documents need to be. Then walk the pipeline: ingestion, chunking strategy, embeddings, hybrid keyword and vector search, re-ranking, a prompt that cites sources, and a fallback when nothing relevant comes back. Cover access control, because retrieval must never surface a document the user cannot open. Finish with how you would evaluate retrieval and answers separately.

3. Users say the assistant gives confident wrong answers. How do you find out why?

Collect real failing examples before touching anything. Split the failure into retrieval (the right passage never arrived) versus generation (it arrived and the model ignored or distorted it). Check traces for each case. Then fix the layer that is actually broken, and add those examples to your eval set so the fix stays fixed. Mention letting the model say it does not know as a legitimate outcome.

4. How would you build an evaluation suite for an LLM feature before launch?

Begin with a small, hand-labelled set drawn from realistic inputs, including edge cases and adversarial ones. Define what good looks like per task: exact checks where possible, rubric-based model grading where not, with a sample of the grader checked by humans. Run it on every prompt, model or retrieval change, and track it over time. Be honest that eval sets go stale, and explain how production traces feed new cases back in.

5. Design an agent that can complete a multi-step task using our internal tools.

Clarify the task and which tools it really needs. Describe the loop: plan, call a tool, read the result, decide the next step, stop. Cover tool schemas with tight inputs, a step and spend budget, clear stopping conditions, retries for flaky tools, and full traces so you can replay a run. Say which actions need human approval. Interviewers want to hear that you have thought about the agent getting stuck or going rogue, not just the happy path.

6. When would you not use an agent?

Most of the time, honestly. If the steps are known in advance, a fixed workflow with an LLM at specific points is cheaper, faster and far easier to test. Agents earn their place when the path genuinely depends on what the model finds along the way. Give an example where you simplified an agent into a workflow, or chose a workflow from the start, and what that saved.

7. Your agent can send emails and update records. How do you stop a prompt injection from abusing that?

Treat everything the agent reads, web pages, documents, emails, as untrusted input. Give tools the least privilege they need, scope credentials to the user, require confirmation for irreversible or external actions, and separate reading untrusted content from taking actions where you can. Log and alert on unusual tool use. Admit that no prompt-level defence is complete, which is exactly why the permission design matters.

8. Our LLM costs tripled last month. What do you do?

Find out where the tokens go before cutting anything: which feature, which prompts, input versus output, retries, runaway agent loops. Then work through the levers: trim context, cache repeated prompts and results, route easy requests to a smaller model, cap agent steps, and batch work that does not need to be real time. Check every change against your evals so you do not trade cost for quality without noticing.

9. When would you fine-tune a model rather than improve the prompt or retrieval?

Fine-tuning helps with consistent format, tone or a narrow task at high volume, or to get a smaller model doing what a larger one does. It is a poor way to add facts that change, which is what retrieval is for. Say you would exhaust prompting and retrieval first, measure the gap with evals, and only then fine-tune, knowing you now own data pipelines and re-training when the base model changes.

10. Tell me about an LLM feature you shipped that did not work as expected.

Pick a real one. Describe what you expected, what users actually did, and how you found out, ideally from traces or evals rather than an angry email. Explain the fix and the process change that came with it. This question is checking whether you learned to measure, so the lesson should be about evaluation or monitoring, not just a better prompt.

11. Describe a time you had to explain the limits of an AI feature to a non-technical stakeholder.

Show you can be straight without being gloomy. Name what they wanted, what the model could reliably do, and how you showed them the gap, ideally with real examples rather than a lecture on hallucination. Say what you agreed instead, perhaps a narrower scope or a human review step, and how it went after launch.

12. How do you keep up with a field that changes every month without chasing every new model?

Describe a filter rather than a reading list. For example, you test new models against your own eval set before forming an opinion, and you only switch when the results justify the migration cost. Mention one recent change you adopted and one you deliberately ignored, and why. That shows judgement, which is the actual skill being tested.

How to prepare

Say each answer out loud, keep it short, and swap in an example from the job you are actually chasing.

• Bring one system diagram of an LLM feature you built: data in, retrieval, model calls, tools, evals, monitoring. Hang your answers off it.
• Have real numbers ready from your own work, such as latency, cost per request or eval scores before and after a change. Do not invent them.
• Prepare one agent story and one story where you chose not to build an agent. Interviewers increasingly ask for both.
• Practise explaining prompt injection and eval design to someone non-technical. If you cannot, you do not understand them well enough yet.
• Run the RAG design question out loud in a Voxxhire mock. It always takes longer to say than you think.