LLM System Design Interviews: 36 Reported Prompts (2026)

LLM and generative AI system design interviews ask you to design a production system around a model, and the prompts candidates write about most are serving and infrastructure problems rather than chatbot features. As of 2026-10-16, TrueInterview's bank holds 36 LLM system design questions reported at 16 big-tech companies. The most-linked prompt is Anthropic's Design GPU Inference Serving System, cited in 22 candidate write-ups, while Design AI Chatbot App and Design a RAG System are linked from 2 write-ups each. Our take: this round is an infrastructure and reliability exercise built on distributed-systems fundamentals, so prepare serving, batching, caching and the trust boundary around the model before you add another RAG component to your notes.

Disclosure: TrueInterview is an interview-preparation product and publishes this article. Facts about other products come from their public pages on the dates listed under Sources.

What do LLM system design interviews actually ask?

They ask you to design a production system with a model inside it, and most of the reported prompts sit in one problem family. Of the 36 questions, 27 are filed as ML system design, 7 as distributed systems, 1 as frontend system design and 1 as experiment design. By problem family, 28 fall under LLM, RAG and AI infrastructure and 4 under ads, recommendation and ranking ML.

The company spread is wide but uneven: OpenAI has 8 questions in the bank, Anthropic, Apple and Microsoft 4 each, Databricks and Amazon 3 each, and ByteDance, Google, Meta and NVIDIA 2 each. The set is also current, with 34 of the 36 last reported within the 12 months before 2026-10-16. For your preparation, that means the shape of this round is small enough to cover: one family dominates, and the prompts are recent enough that they describe the loops running now.

These are candidate-reported questions reconstructed for practice, not any company's official question list.

Which prompts have candidates reported, by company and month?

The table below lists ten reported prompts with company, round and month attached, newest first, each with its practice page. Read it as a map of what candidates wrote down after their loops, not a forecast of what a company will ask you.

Several more named prompts carry write-up counts rather than a dated round in our data. Anthropic's Design AI Prompt Playground is linked from 10 write-ups, and its Route and Batch Inference with Eight GPUs from 3. Design AI Chatbot App is filed under OpenAI, Apple, Databricks and Uber, and Design a RAG System is linked from 2 write-ups. The ads, recommendation and ranking family includes Pinterest's Personalized Chatbot Recommending Pins, Snapchat's Design short-video retrieval with sparse text and ByteDance's Dynamic K in Retrieval Stage. Stock Trading Agent System Design is linked from 2 senior-or-above write-ups.

The rest of the set lives in the practice question bank, filterable by company. If you only have time for a few, the decision rule near the end of this article picks them for you.

Our take: is this round about chatbots or AI infrastructure?

Infrastructure, in our reading, and most clearly at AI labs. The most-linked prompt is a serving design, and the application prompts in the dated table name access control, memory and reliability in their titles rather than features. The most-linked LLM-family prompts are Anthropic's GPU inference serving and prompt playground designs, Meta's agent system and Anthropic's eight-GPU routing problem.

Read the link counts with care. The 22 GPU serving links all come from Anthropic write-ups, so the infrastructure lean is strongest for AI-lab loops, not every LLM design round. Outside Anthropic, Meta's agent system is linked from 5 write-ups and Google's mixed ML System Design prompt from 3, while Design AI Chatbot App draws 2. Application prompts still matter at Amazon, NVIDIA and Microsoft, as the dated table shows, but their titles turn on reliability, access control and memory, which are infrastructure questions in product clothing.

What this means for your next week: before you rehearse any assistant or copilot, be able to draw an inference request path end to end. That is request routing, batching across a fixed GPU pool, queueing, and a clear answer for what happens when capacity runs out. An assistant design reuses most of that path, so the infrastructure work pays twice.

How is GenAI system design different from classic ML system design?

The difference is in what you can specify. PracHub's guide puts it as "you must design around a component whose behavior cannot be fully specified in advance". Classic ML designs produce a bounded output you can score; generative designs produce open-ended output, so evaluation and the trust boundary move into the architecture itself.

Verdict for classic ML design: keep it if your target team runs recommendations or ranking, because some reported "chatbot" prompts, such as Microsoft's memory design, are filed in the recommendation family. Verdict for GenAI design: this is the one to over-prepare if your loop is at an AI lab, because the trust boundary and the evaluation box are where it differs, and both are drawable in five minutes once you have practised them.

When should your design use prompting, RAG, tools or fine-tuning?

Start with prompting, add retrieval or tools only when you can name the requirement prompting misses, and reach for fine-tuning last. PracHub's guide warns, "Do not default to fine-tuning before explaining why prompting, retrieval, or tools are insufficient." Our view: the reported prompts fit that order, because most of their titles are about the system around the model.

Our view of the ladder, mapped to real prompts:

• Prompting alone fits when the task needs no private data and the output is easy to check. Our reading of Anthropic's prompt playground title is a system for running and comparing prompts, so its hard parts are versioning, quotas and evaluation runs rather than retrieval.
• Retrieval (RAG) fits when answers depend on data that changes or that the model never saw. The deciding question is freshness against cost: how stale an index you can tolerate, and what you do when retrieval returns nothing useful.
• Tools fit when the model must act or query live systems. ByteDance's agent tool-use prompt and NVIDIA's SQL chatbot sit here, and the deciding question is which actions the model may take without deterministic code checking them first.
• Fine-tuning fits when behaviour, format or latency targets cannot be met by the three options above, and you can afford a training and re-evaluation loop for every base-model update.

If you are senior or above, say out loud which rung you rejected and why. That sentence carries more signal than a longer component list.

Has classic distributed-systems design gone away?

No. Classic distributed-systems design still sits inside these loops, and software engineers' write-up links lean toward it more than ML candidates' do. OpenAI's Payment / Coffee-Shop Ordering prompt is linked from 16 write-ups and Databricks's In-Memory KV Cache with WAL Log from 12, making them the second and third most-linked prompts in the set.

Software engineers see more of these designs than ML candidates do. Payments, ledgers and money movement is 20% of software engineers' write-up links against 6% for machine-learning, AI, research and data-science candidates, and storage, caching and key-value stores is 16% against 0%.

The deciding questions come from standard engineering practice. Our view: for the ordering flow, be ready to defend idempotency keys and a ledger that makes double charges impossible to miss. For the KV cache, be ready to defend your fsync policy, how you replay the write-ahead log after a crash, and when you snapshot and truncate it. The OpenAI title's own read-the-prompt warning is the cheapest advice here: confirm the requirements before you draw.

Do these prompts appear in phone screens or only onsites?

Both. Of these questions, 22 were reported in an onsite and 17 in a phone screen, and a question can be reported in both. Of the 17 phone-screen-reported questions in this set, 13 are in the LLM, RAG and AI infrastructure family, so LLM design prompts are not reserved for the onsite.

The onsite skews the same way, with 16 of 22 questions in that family and 3 in ads, recommendation and ranking. Two of the dated prompts above, Anthropic's GPU serving design and Amazon's reliable chatbot, were reported in the phone screen.

PracHub's guide says these rounds usually give you about 35 to 60 minutes to define the problem, propose an architecture, go deep on one or two components and explain trade-offs. Our view: prepare a compact version of your serving and chatbot designs before the screen, not after it, because a phone-screen slot leaves no room to discover your structure live.

Who gets these questions, by level and role?

Mostly senior software engineers. Candidates who reported a senior, staff-plus or manager level link these questions 81 times, against 12 links from those who reported junior, new-grad or intern. By role, software engineers account for 71 links and machine learning engineers for 13, so this is not an ML-only round.

The infrastructure prompts are the most senior. GPU inference serving, the Meta agent system and the eight-GPU routing problem are linked only from senior-or-above write-ups. The KV cache with WAL is the most mixed prompt, with 7 senior-or-above links and 5 junior-or-below, which makes it a fair prompt for any level and a good warm-up.

Role changes the emphasis more than the topic. GPU inference serving is linked from 19 software-engineer write-ups and 2 from machine-learning, AI, research or data-science candidates. If you are a software engineer, the serving prompt is yours to prepare, not a specialist's.

What will a senior interviewer push on?

Our view: prepare to defend the eval set, the fallback path, and cost against latency. PracHub's guide names the root mistake: "treating the large language model as the entire system". PracHub's guide lists context pipelines, orchestration, evaluation, safety controls, observability, failure recovery and latency and cost control as what a production product needs. These three decisions are where that list turns into trade-offs.

The eval set

Our view: describe a fixed, versioned set of inputs with expected properties, run on every prompt or model change, plus sampled live traffic reviewed by a grader. An arXiv paper on production LLM failures notes that existing benchmarks "provide little insight into stability, reproducibility, drift, or workflow integration", so a public benchmark score is not your eval plan. The arXiv paper also names version drift and update-induced regressions among production challenges, which is why the eval set has to gate model upgrades.

The fallback

Decide what the user sees in three cases: the model times out, returns an answer your validator rejects, or is unavailable. For the Amazon reliable chatbot prompt, that means timeouts, a smaller backup model or a cached answer, and a clean handoff when confidence is low. For the NVIDIA SQL chatbot, the fallback is refusal: a read-only database role, permissions enforced in the database, and a validator that rejects a generated query before it runs.

Cost and latency

State a latency target per request type and a cost budget per request before you draw. Batching raises GPU throughput but adds queueing delay; streaming hides generation time but not time to first token. The arXiv paper lists cost-driven performance collapse among its named failure modes, so say how your design degrades when the budget binds: shorter context, a cheaper model, or shed load.

Classic follow-up pressure still applies to these rounds. One candidate's Stripe onsite write-up on TrueInterview reports a rate limiter that began as a sliding window, then had to be redone with a shared store across three app servers. Other reported follow-ups ask how to keep a design correct under clock skew and what breaks first at ten times the traffic. Those come from classic design rounds, not LLM ones, but they are worth rehearsing against your own LLM gateway.

Worked example: design GPU inference serving

Take Anthropic's Design GPU Inference Serving System, reported in the phone screen and linked from 22 write-ups by senior-or-above candidates, 2 of them staff-plus or manager. A classic ML answer would spend its time on training pipelines and feature stores. That is the wrong half of this problem, so spend your minutes on the request path.

1. Frame it. Ask who calls the API, the latency target, and whether responses stream. Two minutes here decides every later trade-off.
2. Draw the path. A gateway with per-tenant rate limits, a router, a batching layer feeding a fixed GPU pool, and a streaming response channel.
3. Go deep on batching and routing. This is the hard part, and Anthropic's Route and Batch Inference with Eight GPUs prompt is linked from 3 senior-or-above write-ups. Explain how continuous batching admits new requests as others finish, how GPU memory for the attention cache limits batch size, and how the router balances load without splitting a request across GPUs.
4. Handle exhaustion. Bound the queue, reject early with a retry hint, and prioritise by tenant so one caller cannot starve the rest.
5. Close on production. What you monitor (queue depth, time to first token, GPU utilisation), how you catch regressions after a model version change, and how a cost cap changes routing.

For contrast, Databricks's onsite KV cache with WAL is pure distributed-systems design in the same set. Practise both in the same week, so a loop that pairs a GPU serving prompt with a storage prompt does not catch you in the wrong mode.

What changes at senior and staff level?

The bar moves from naming components to defending trade-offs under failure. By the levels candidates reported, there are 81 links from senior-or-above candidates against 12 from junior, new-grad or intern candidates, and GPU inference serving, Meta's agent system and the eight-GPU routing prompt appear only in senior-or-above write-ups. Our take: a senior answer is graded on capacity limits, model version changes and cost, not on the boxes.

At staff level, our view is that interviewers expect you to set the scope yourself: which tenants matter, which failures are acceptable, and what you would build first if you had one quarter. Treat the eval set and the fallback as your own decisions to propose, not questions to wait for.

Down-leveling note (our view): in these loops, design depth carries most of the level signal. A clean architecture with no failure section reads as mid-level. If you are targeting senior, make sure your design round includes one explicit trade-off you rejected, one failure you planned for, and one number you committed to.

Which prep plan fits your loop?

Pick the plan by company type and role. AI labs and AI-infrastructure companies call for serving and batching first; product companies call for one LLM application design plus classic storage and payments; ML engineers lean further toward LLM, RAG and AI-infrastructure prompts. In every case, assume a design slot can appear in the phone screen.

• AI lab or AI-infrastructure company, senior or above: spend most of your design time on inference serving and batching, then one classic storage or ordering design.
• Software engineer at a product company: one LLM application design with memory, access control and evaluation, plus classic storage, caching and payment designs.
• ML engineer: lean further toward LLM, RAG and AI-infrastructure prompts, which take 82% of write-up links from machine-learning, AI, research and data-science candidates against 58% from software engineers.

The hope here is honest: 36 questions, 28 of them in one family, with a handful of prompts drawing most of the write-ups, is a narrow target. A small, repeatable set of designs covers the shape of these rounds.

How we counted

Counts come from TrueInterview's question bank as of 2026-10-16, where each candidate-reported question is filed by company, round, month and the candidate's self-reported level and role. Problem families are assigned from question titles, and a question can be reported in more than one round. Write-up link counts show how many candidate write-ups mention a question, not how often a company asks it, and level splits rely on what candidates reported about themselves.

FAQ

Are there free GenAI system design question lists on GitHub or Reddit?

A free list is most useful when it states where each prompt came from. A list of reported prompts with company and month, like the dated table above, tells you more about what to practise. Use free lists for vocabulary and use reported prompts to choose what to draw. If a list does not say where a prompt came from, treat it as a writing exercise.

Is a GenAI system design book or PDF worth reading?

A book helps with building blocks such as retrieval, caching and evaluation, and is worth one pass if those terms are new to you. It will not tell you which prompts candidates report at which company. Our view: read for vocabulary, then spend most of your hours drawing reported prompts against a timer, because the round grades your trade-offs, not your recall of a chapter.

Do junior candidates get LLM system design rounds?

Some do, but far fewer report it. Junior, new-grad and intern candidates account for 12 links in this set, against 81 from senior-or-above candidates, and the GPU serving, Meta agent system and eight-GPU routing prompts have no junior links. Our view: if you are junior, the KV cache with WAL, which has 5 junior-or-below links, and Anthropic's Design AI Prompt Playground, which has 2, are a better use of your time than GPU serving.

How long is the round, and how should you split the time?

PracHub's guide gives a range of about 35 to 60 minutes. Our view of the split: about a tenth on requirements, a third on the high-level path, a third on one deep dive, and the rest on failure modes, evaluation and cost. Cut the deep dive, never the failure section, when you run short.

Your practice plan

A three-week plan for someone working full time: two or three evening sessions a week, each one prompt drawn end to end and defended aloud.

1. Week one, first session: write a one-page design for Design GPU Inference Serving System, including what you monitor and how you shed load when the GPU pool is full.
2. Week one, later: add the hard part with Route and Batch Inference with Eight GPUs, and explain your batching and routing out loud against a timer.
3. Week two: practise In-Memory KV Cache with WAL Log against a timer, covering crash recovery and log compaction.
4. Week two, later: work Payment / Coffee-Shop Ordering (read the prompt!) with idempotency keys and a ledger, after re-reading the prompt twice.
5. Week three: draw LLM SQL Chatbot Access Control and Query Safety and mark every decision as deterministic code or model output, with an evaluation box in the diagram.
6. Week three, later: do Design a Reliable AI Chatbot Architecture as a phone-screen rehearsal, with an explicit fallback path and a cost cap.
7. Before the loop: filter the practice question bank by your target company and redo the one prompt you found hardest.

Sources

• TrueInterview question bank — LLM system design at big-tech companies — counted 2026-10-16
• Generative AI System Design Interviews: How to Design LLM Applications That Work in Production | PracHub Knowledge Hub — checked 2026-10-16
• 2511.19933 Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications — checked 2026-10-16
• Interview prep compared: your application, end to end · TrueInterview — checked 2026-10-16

Last reviewed: 2026-10-16.