The Interview Process
Stage 1: Recruiter screen (30 min)
CV walkthrough, motivations, salary expectations, and a light culture check. Prepare a 2-minute summary of your most relevant AI project.
Stage 2: Technical screen (45–60 min)
Python fundamentals, basic ML concepts, and possibly one LLM API question. Often asynchronous via a platform like Codility or HackerRank.
Stage 3: Take-home or live coding (1–3 hrs)
Build a small AI feature — often a RAG pipeline, an API endpoint wrapping an LLM, or a data processing task. Clarity and testing matter as much as the working solution.
Stage 4: System design (45–60 min)
Design an end-to-end ML or LLM system at production scale. Focus on monitoring, evaluation, and failure modes rather than just the happy path.
Stage 5: Behavioural / culture (45 min)
STAR-format questions about past projects, collaboration, and handling failure. Senior roles will also probe engineering judgment and strategic thinking.
Technical Questions
Read the question, write your own answer first, then compare against the example response.
Q1. How would you design a RAG (retrieval-augmented generation) pipeline for a customer support chatbot?
Strong answer
Start by describing the full pipeline: document ingestion → chunking strategy (e.g. 512-token chunks with 50-token overlap) → embedding with a model like text-embedding-3-small → storage in a vector database (Pinecone, Qdrant, or pgvector). At query time: embed the user query → retrieve top-k chunks by cosine similarity → pass them as context to the LLM with a structured prompt. Highlight trade-offs: chunk size affects recall vs. context window usage; reranking (e.g. cross-encoder) improves precision but adds latency. Mention evaluation: measure faithfulness and answer relevance with RAGAS or a custom eval harness.
Q2. What strategies do you use to reduce LLM latency in a production API?
Strong answer
Key strategies: (1) Streaming responses so the user sees tokens as they arrive rather than waiting for the full completion. (2) Caching — semantic caching (e.g. GPTCache) for near-duplicate queries, exact caching for deterministic prompts. (3) Model selection — use a smaller/faster model (e.g. GPT-4o Mini, Claude Haiku) for simple tasks and route complex queries to the capable model. (4) Prompt compression — trim unnecessary context. (5) Speculative decoding for self-hosted models. Always back answers with latency budgets: what's acceptable (200ms for autocomplete vs. 3s for summarisation).
Q3. Explain how you would evaluate an LLM-based feature before shipping it to production.
Strong answer
I'd build a layered evaluation approach. First, offline evals: a curated dataset of golden examples with expected outputs, scored against criteria like factual accuracy, tone, and task completion using both model-based judges (e.g. GPT-4 as evaluator) and rule-based checks. Second, A/B testing in production using metrics that matter to the user (resolution rate for support, click-through for recommendations). Third, ongoing monitoring: logging prompts and completions, tracking refusal rates, latency, and token cost. Avoid shipping without a baseline and at least one human review pass.
Q4. What is the difference between fine-tuning and prompt engineering, and when would you choose each?
Strong answer
Prompt engineering is zero-cost to deploy, fully reversible, and should always be tried first — for behaviour shaping, output formatting, and task decomposition. Fine-tuning is appropriate when: (1) you need consistent stylistic or domain-specific behaviour that can't be reliably achieved via prompting, (2) you need to reduce token cost at scale (a fine-tuned smaller model can outperform a prompted larger one), or (3) you need to reduce prompt length in latency-sensitive applications. Fine-tuning requires labelled examples, compute, and ongoing maintenance — it's a commitment, not a quick fix.
Q5. How do you handle hallucinations in a production LLM system?
Strong answer
Hallucinations can't be eliminated, only managed. Defence-in-depth: (1) Ground responses in retrieved context (RAG) so the model has a source to cite. (2) Add output validation — structured outputs (JSON mode, function calling) prevent format hallucinations. (3) Self-consistency: run the same query multiple times and compare; flag divergent answers for human review. (4) Confidence signalling in the prompt: 'if you are not confident, say so'. (5) Downstream guardrails: check that cited sources actually exist, validate numerical claims programmatically. Monitor refusal rates and user feedback loops.
Q6. Walk me through how you'd build a model serving infrastructure for a high-traffic inference endpoint.
Strong answer
I'd start with the model server: vLLM or TGI for LLMs (PagedAttention for efficient GPU memory), Triton Inference Server for traditional ML models. Layer a load balancer (NGINX or a cloud ALB) in front of multiple instances. For scaling: horizontal pod autoscaling in Kubernetes triggered by GPU utilisation or queue depth. Add a request queue (Redis or Celery) to absorb traffic spikes without dropping requests. Cache frequent responses at the application layer. Instrument everything: p50/p95/p99 latency, tokens/second, GPU utilisation, error rate. Define SLOs before launching.
Q7. What metrics would you use to monitor an AI feature after deployment?
Strong answer
I separate three types: (1) Technical health — latency (p50/p95), error rate, token cost per request, model uptime. (2) Feature performance — task-specific metrics (resolution rate for support, accuracy for classification, BLEU/ROUGE for summarisation). (3) Business impact — downstream KPIs that the feature was meant to move (conversion, engagement, support ticket deflection). Also monitor for data drift: if input distributions shift, performance may degrade silently. Set up alerts on p95 latency and error rate, review feature metrics weekly.
Q8. Describe a situation where you had to debug a non-deterministic bug in an AI pipeline. How did you approach it?
Strong answer
Non-determinism in LLM pipelines comes from temperature > 0, network failures, or upstream data drift. My approach: first, reproduce the bug with a fixed seed and temperature=0 to isolate whether it's stochastic or deterministic. Log every step with unique request IDs so I can trace the full path of a failing request. Add output schema validation at each stage so I catch malformed outputs early rather than at the final step. If the bug is stochastic, build a regression suite of adversarial inputs and run it on every deploy.
Q9. How do you approach prompt injection risks in a customer-facing LLM product?
Strong answer
Prompt injection is where user input manipulates the system prompt or causes the model to act outside its intended role. Defences: (1) Clearly delimit user input from system instructions (e.g. XML tags, separate message roles). (2) Validate and sanitise inputs — strip special characters that could break prompt structure. (3) Use a secondary safety classifier (e.g. Llama Guard) to screen inputs and outputs. (4) Never put sensitive instructions (API keys, internal logic) in the system prompt where they could be extracted. (5) Apply principle of least privilege: the model should only have access to tools it actually needs.
Q10. How do you keep up with the pace of change in AI tooling and research?
Strong answer
This question tests learning habits and signal/noise filtering. Good answer: follow a small set of high-quality sources (Hugging Face blog, LangChain release notes, specific Substack authors, arXiv Sanity for papers). Allocate time each week to read and experiment — ideally building a small proof of concept rather than just reading. Mention participating in the community: writing about experiments, contributing to open source, attending local meetups. Avoid 'I read everything' — the ability to triage and focus is what matters in a fast-moving field.
Behavioural Questions
Use the STAR method (Situation, Task, Action, Result) and keep answers to 2–3 minutes.
Tell me about a time an AI feature you built didn't work as expected in production. What did you do?
Show you monitor proactively, diagnose systematically, and communicate clearly with stakeholders. Avoid blaming the model.
Describe how you've worked with non-technical stakeholders to define requirements for an AI system.
Demonstrate that you translate between technical constraints (what models can/can't do) and business needs. Mention how you set expectations around uncertainty.
How do you decide when a problem is worth using AI for vs. a simpler rule-based approach?
Show engineering judgment. AI adds complexity, cost, and unpredictability. A good engineer reaches for it only when the problem genuinely benefits.
Walk me through a technical decision you pushed back on. What was the outcome?
Demonstrates confidence, communication, and intellectual honesty. Pick an example where you were right — and one where you were wrong and learned from it.
How do you balance moving quickly on AI experiments with maintaining code quality?
Shows pragmatism. Mention the difference between prototype code (fast, throwaway) and production code (tested, reviewed, documented). Describe how you draw the line.
Red Flags to Watch For
Questions to ask — and signals that suggest the role or team may not set you up for success.
No model monitoring or alerting
If the team can't describe how they detect when a model degrades in production, you'll be flying blind after every deploy.
"We'll add evals later"
Evaluation is not optional — it's how you know if you're making things better or worse. Teams that defer it rarely add it.
No clear ownership of the AI system
If it's unclear who is responsible for model quality, latency, and cost, those things won't be managed well.
AI is being added without a clear user problem
Technology-led product decisions lead to low-impact work. Ask what metric the AI feature is supposed to move and whether there's a baseline.
No documentation of prompt changes
Prompts are code. If they're not versioned and reviewed, regressions will be invisible.