AI system design
The curriculum and the questions. The teaching lives in llm-lab — every row below names the lesson
that covers it, so nothing is duplicated and nothing has two versions to keep true.
ENTRY for AI Engineer roles. EDGE at your backend targets.
Gating at AI and GenAI engineer roles, which is where most of the new headcount is. Not yet a gating round at Uber, Atlassian, Salesforce, Microsoft, Visa or Amazon — there it is a differentiator rather than a requirement. Either way it does not replace DSA: these loops keep a classic coding round, and Anthropic, OpenAI, Scale and Sierra ask their AI-infrastructure questions inside SWE and ML rounds. There is no fork. This layer is additive.
An open-ended 35 to 60 minute conversation, not a whiteboard algorithm. Almost every 2026 loop has at least one deep RAG round, and agents now get a full round of their own. Interviewers no longer want the textbook definition of an embedding — they want to know what breaks, what you have shipped, and how you reason when there is no clean answer.
The seven pillars, tested as a set
RAGRetrieval quality, chunking, hybrid search, reranking, and the failure taxonomy when an answer is wrong.
Fine-tuningWhen it beats prompting and retrieval, what it costs, and what it cannot fix.
AI cachingPrompt caching, semantic caching, result caching — and what each invalidates on.
Agentic evalsScoring a system that is non-deterministic and fails a third of the time by design.
AI observabilityKnowing exactly what happened inside one request, end to end.
User feedback loopsTurning thumbs-down into an eval set rather than a dashboard nobody reads.
AI guardrailsBounding autonomy: stopping conditions, cost caps, and refusing a destructive action.
1 · Foundations you need before anything else
Short. Skip what you already have from llm-lab Block 1.
The model call as a system component
Slow, expensive, flaky, rate-limited, non-deterministic. Every downstream design decision follows from those five properties.
llm-lab · lc.initchat, lc.messages
Tokens, context windows and cost arithmetic
Be able to compute the cost of a request out loud, in the room.
llm-lab · lc.generation
Structured output and validation at the boundary
What to do when the model returns something that does not parse.
llm-lab · lc.generation, lc.tools
Embeddings: cosine, dot, L2, normalisation
Enough to reason about retrieval, not enough to train one.
llm-lab · emb.vectors, emb.local
2 · RAG, in the order it is asked about
The highest-value block. Almost every loop has a deep RAG round.
The pipeline and its failure taxonomy
Ingest, chunk, embed, index, retrieve, rerank, assemble, generate. Know which stage each failure mode belongs to, and the order you diagnose in.
llm-lab · rag.anatomy
Chunking strategy
Size, overlap, augmentation, decoupled retrieval and generation units. The most common cause of bad answers that look like model failures.
llm-lab · rag.chunking
Hybrid search and reciprocal rank fusion
Why pure vector search loses on exact terms, identifiers and rare words.
llm-lab · rag.fusion (read rag.bm25 first)
Reranking
What a cross-encoder buys you and what it costs in latency.
llm-lab · rag.fusion
Evaluating retrieval separately from generation
Recall@k on the retriever before you blame the model. Most teams never do this.
llm-lab · rag.anatomy
Building the whole thing in ~150 lines
When you can write it from memory the subject stops being intimidating.
llm-lab · rag.scratch
3 · Agents
A full round of its own now. The hardest section in 2026 because companies want people who have seen what breaks.
The loop, in plain code first
Perceive, decide, act, observe, repeat. Write it without a framework before you touch one.
llm-lab · lg.loop
When a graph beats a chain
Branching, retries, and state that outlives a single call.
llm-lab · lg.why, lg.stategraph
Tools: schema, validation, and failure
The docstring is the schema. What happens when a tool returns garbage, or times out.
llm-lab · lc.tools
Memory
Short-term context versus durable state, and what you are willing to lose on a crash.
llm-lab · lg.stategraph
Bounded autonomy
Stopping conditions, step limits, cost caps. An unbounded agent is a billing incident.
llm-lab · lg.bounds
Safe action: dry-run, confirm, idempotent execute
The three-step pattern that makes an agent safe to point at real systems. This is also the prompt-injection answer.
llm-lab · lg.safeaction
Agentic RAG
The agent decides when to retrieve and how to reformulate, rather than retrieving once up front.
llm-lab · lg.agenticrag (skipped in the fast track)
4 · Production: observability, cost, evaluation
Where a backend engineer has an unfair advantage, because this is ordinary systems work wearing a new hat.
The observability chain
traces → spans → metrics → evals → debugging → cost. You must be able to narrate one request through all of it: prompt construction, retrieval, reranking, tool calls, model calls, guardrails, token usage, latency, errors, output quality.
llm-lab · obs.spans
Cost control
Retry storms, context windows quietly filling with retrieved junk, model routing, caching, token budgets per request.
llm-lab · lc.production
Evaluation
An eval set, an LLM-as-judge whose own blind spots you can name, and regression tests on prompts. Repeatable, not vibes.
llm-lab · lc.production, rag.anatomy
Reliability
Retries, fallbacks, circuit breakers, graceful degradation when the provider is down.
llm-lab · lc.production
Serving
Continuous batching — finished sequences leave the batch, new ones join mid-generation. Say that phrase specifically; it distinguishes you from generic request batching. Plus latency budgets, throughput, GPU utilisation.
llm-lab · srv.design (skipped; worth revisiting)
Quantization
What you trade away, and which formats. Asked most at inference and ML-infra roles.
llm-lab · not in the fast track
5 · Safety and guardrails
Prompt injection
A retrieved document is untrusted input. Treat it the way you treat a request body.
llm-lab · lg.safeaction
Output filtering and refusal paths
What the system does when the model produces something it should not act on.
llm-lab · lg.bounds
Data boundaries
What leaves your infrastructure, what is logged, and what a trace must never contain.
llm-lab · obs.spans
What they actually ask
Production-failure shaped, never definitional. Every one of these is a real reported question.
Your LLM costs jumped 300% overnight with no traffic increase. Investigate.
what they wantRetry storms. A context window quietly filling with retrieved junk. A prompt template change. Model routing silently falling back to an expensive model. And the honest admission that nobody built the dashboard that would have caught it in an hour.
Debug a RAG system that hallucinates in production.
what they wantDiagnose in order: is the right chunk being retrieved at all (recall@k), is it surviving reranking, is it inside the assembled context, is the model ignoring it. Most people jump straight to prompt fixes for a retrieval bug.
Evaluate an agent that fails 30% of the time.
what they wantAn eval set built from real failures. Trajectory evaluation, not just final-answer scoring. An LLM-as-judge whose blind spots you can name. Regression tests so a prompt change cannot silently undo a fix.
Stop a prompt-injected document from triggering a destructive action.
what they wantDry-run, confirm, idempotent execute. Treat retrieved content as untrusted input. Separate the plan from the execution and gate the gap.
Architect an AI agent system: the loop, tools, memory, safety.
what they wantDraw it. Defend each boundary. Name what happens when a tool times out mid-plan and when the process dies between steps.
Design a RAG pipeline, and defend the eval methodology.
what they wantThe methodology is the real question. Anyone can draw the pipeline.
How would you serve this at scale?
what they wantContinuous batching, latency budget, caching layers, GPU utilisation, and where you would degrade first under load.
What is quantization and what do you trade away?
what they wantAccuracy against memory and latency, and which format for which constraint.
Why this is an edge for you specifically
This round rewards someone who has run production systems and is comfortable with failure modes, retries, idempotency, observability and cost — which describes a backend engineer, not a prompt engineer. You have shipped idempotent payments, Redis distributed locks, DLQ monitoring with reconciliation, and a profiling win you can put a number on. Almost nobody walking into an AI loop has that, and almost nobody walking into a backend loop can debug a hallucinating RAG pipeline. The combination is the scarce thing. The gap is that payments-ops-agent was built in three hours by an agent and you cannot yet defend it — llm-lab Block 6 closes exactly that, in about three hours of reading your own repo.
Who asks this
Reported from real loops at OpenAI, Anthropic, Scale AI, Sierra, xAI, Databricks and Perplexity. Note how they run it: Anthropic, OpenAI, Scale and Sierra fold the AI-infrastructure questions INTO their SWE and ML rounds rather than running a separate AI round, so engineering fundamentals and AI topics are prepared together. Meta replaced Product Architecture with AI-Enabled Product Architecture on 16 April 2026. Google's code-comprehension pilot puts Gemini in the room and scores AI fluency directly. At the rest of your target list this is upside, not a gate.
Sources
Agentic AI system design interview guide 2026 · Design an agentic AI system — a new class of question · AI engineering interview prep: observability and tracing · Generative AI engineering interview 2026 · AI engineer interview questions 2026 · RAG interview questions 2026
The AI system design round: the curriculum and what gets asked. Deliberately NOT the lesson content — that lives in llm-lab, and duplicating it here would create a second copy to keep true. Researched 2026-09-25, 2026 sources only.