GenAI System Design Interview Prep: Start With GPU Serving

As of 2026-11-02, TrueInterview's bank holds 28 candidate-reported GenAI and AI infrastructure design questions from 13 big-tech companies. The two most-linked are infrastructure problems: Design GPU Scheduling Platform, linked from 28 write-ups, and Design GPU Inference Serving System, linked from 26, while Design AI Chatbot App is linked from 5. For a loop at Anthropic, OpenAI or Databricks, where those GPU titles come from, prepare the serving and scheduling layer first, meaning KV-cache memory, request batching and GPU allocation, and treat the chatbot or retrieval design as the second, smaller target.

Disclosure: TrueInterview is an interview-preparation product and publishes this article. Facts about other products come from their public pages on the dates listed under Sources.

Which GenAI design prompts do candidates report, and where?

Candidates report infrastructure and application prompts side by side, spread thinly across companies: OpenAI files 6 questions, Anthropic 5, Apple and Microsoft 4 each, Amazon and NVIDIA 3 each, Databricks and Airbnb 2 each, and Uber and Netflix 1 each. The table below shows one dated example per company; filter the practice bank by your target company to see the rest of its list.

By problem family, 24 of the 28 questions are LLM, RAG and AI infrastructure designs. By subtype, 18 are ML system design and 6 are distributed systems. The list is current: 25 of the 28 were last reported in the 12 months before 2026-11-02. That recency is what lets you use the table below as a syllabus rather than as a history of the format.

These are candidate-reported questions reconstructed for practice, not any company's official question list.

Our take: by title alone, the table looks evenly split between GPU fleet problems and chatbot problems. Titles are the wrong unit for ordering your preparation. The write-up links in the next section show where candidates keep landing, and they point at the serving and scheduling layer.

Why should GPU serving and scheduling come before chatbot designs?

Because the two GPU titles in the opening paragraph are each linked about twice as often as the most-linked application prompt, Anthropic's Design AI Prompt Playground, and more than five times as often as Design AI Chatbot App. Which two designs to build to full depth depends on the company: for Anthropic, the inference server and the prompt playground; for OpenAI or Databricks, the GPU scheduler and the chatbot app.

Write-up links count candidate reports. They do not measure how often a company asks a prompt, and the companies a write-up came from can differ from the companies a question is filed under. The notes below say what each title asks for, read from the title, and which decisions an interviewer will push on at senior level. They are general engineering knowledge, not reports of what any interviewer asked.

Design GPU Scheduling Platform

The task is a shared scheduler that places training and inference jobs onto a fleet of GPUs. The first decision is all-or-nothing placement for multi-GPU jobs, so a job never holds half its GPUs while it waits for the rest. The second is preemption with per-team quotas, so a long training run cannot starve serving traffic. Expect to defend how a preempted job resumes: checkpointing and requeueing, not a restart from zero.

Design GPU Inference Serving System

This is the prompt to over-prepare. The decisions an interviewer will push on are how requests are batched on each GPU and how much KV-cache memory each in-flight request holds, because together they set throughput and time to first token. Be ready to say what happens when requests arrive faster than GPUs free up: admit fewer, shed load with a clear error, or route overflow to another replica. Then explain how streaming tokens back to the client changes your timeouts and retries.

The two Anthropic serving drills

Route and Batch Inference with Eight GPUs and Review and Scale an Inference Service Design work as drills on the serving design, not as new topics. The eight-GPU title is a capacity exercise: how requests spread across a fixed pool, how long a batch waits to fill before it runs, and what happens when one GPU drops out. The review-and-scale title asks you to critique a design you did not draw, so practise finding the first bottleneck in someone else's diagram.

Design AI Prompt Playground

This is Anthropic's product-facing title and the one to add after serving. The decisions are how you store prompt versions and run history so users can compare outputs across models, and how you enforce per-user rate limits and cost caps while still streaming responses.

A worked example for an Anthropic loop: build the inference server first, then the eight-GPU drill as a capacity exercise, then the review-and-scale critique as rehearsal for scale follow-ups, and the prompt playground last as the product design. Design GPU Scheduling Platform is filed under Anthropic in the bank, but its write-ups come from OpenAI and Databricks candidates, so for Anthropic it is an optional fifth design if time remains.

What serving mechanics should you be able to explain?

You should be able to explain why the KV cache can take a large share of GPU memory during generation, and how block-based allocation fits more requests into a batch. Hugging Face's text-generation-inference documentation says that during decoding, attention keys and values for previous tokens are stored in GPU memory for reuse, and that this KV cache can take a large amount of memory for large models and long sequences.

The same Hugging Face page describes PagedAttention as partitioning the KV cache into blocks accessed through a lookup table, so the cache need not sit in contiguous memory and blocks are allocated as needed. The Hugging Face documentation says that memory efficiency can increase GPU utilization on memory-bound workloads, so more inference batches can be supported. The page adds that for parallel sampling, where several outputs are generated for the same prompt, cached KV blocks can be shared among the generations.

Turn that into interview reasoning. Each in-flight request holds cache that grows with every generated token, so the number of requests a GPU can batch is bounded by memory before it is bounded by compute. Reserving one contiguous region per request at its maximum length wastes the unused tail; allocating small blocks on demand reclaims it, which raises batch size and therefore throughput. Then name the cost: long sequences still crowd out other requests, so admission control and a per-request token limit belong in the design.

Do not present the original paper's mechanism as current vLLM internals. vLLM's documentation says its PagedAttention design page is a historical document based on the original vLLM paper, and the page states, "It no longer describes the code used in vLLM today."

Our take: explain the idea and its trade-off, not a framework's current code. A senior interviewer is testing whether you can reason from memory per request to batch size to cost per token; naming a library is no evidence of that.

How should you approach chatbot, RAG and memory prompts?

Start with the simplest design that meets the stated requirement, and add a component only when a requirement forces it. Anthropic's engineering post on building agents says, "we recommend finding the simplest solution possible, and only increasing complexity when needed". The same Anthropic post says that for many applications, optimizing single LLM calls with retrieval and in-context examples is usually enough.

Anthropic's post describes the basic building block of agentic systems as an LLM enhanced with retrieval, tools and memory. That is a sound first diagram for Design AI Chatbot App, Design a Reliable AI Chatbot Architecture and Chatbot Personalization / Memory System Design. The Anthropic post also describes routing easy or common questions to smaller, cost-efficient models, and a guardrail in which one model instance processes the query while another screens it for inappropriate content. Add routing when the interviewer sets a cost target and the guardrail when they raise content safety, and say which requirement justified each.

For retrieval, Pinecone's chunking guide recommends starting with fixed-size chunking and iterating only after finding it insufficient. The Pinecone guide adds that large chunks may increase latency and cost in downstream responses, and that long-context models suffer from a lost-in-the-middle problem, where relevant information buried in long documents is missed. Pinecone's guide also warns that tokens beyond an embedding model's context window may be truncated before the text becomes a vector.

In our view, the decisions worth preparing at senior level in these prompts are retrieval quality and access control; no report here says what interviewers pushed on. Measure retrieval separately from generation with a small labelled set of questions and expected passages, so you can tell a retrieval miss from a model error. Filter retrieved documents by the user's permissions before they reach the prompt, because a document the user cannot open must never surface in an answer. For the memory prompt, decide what is stored per user, how a user deletes it and how stale memories are retired. For grounded generation, as in Siri Grounded Response Generation, have the answer point to the passage it relies on and decline when no passage supports it.

If the interviewer pushes toward agents, use Anthropic's distinction: its post defines workflows as systems where LLMs and tools are orchestrated through predefined code paths, and agents as systems where LLMs dynamically direct their own processes and tool usage. Anthropic's post says agentic systems often trade latency and cost for better task performance, and that frameworks can add layers of abstraction that obscure prompts and responses and make them harder to debug.

Our take: reaching for multi-agent orchestration in a chatbot round is an easy way to lose time. Draw the workflow, state the agent version as an extension, and spend the saved minutes on retrieval evaluation and access control, which are the parts a generic answer skips.

Will the design prompt come at the phone screen or the onsite?

Both. Among these questions, 18 were reported in onsite rounds and 13 in phone screens, and one question can appear in both. Of the 18 onsite questions, 16 are LLM, RAG and AI infrastructure designs. Do not treat the phone screen as a lighter round that only carries a warm-up design.

The phone-screen reports include infrastructure: Anthropic's Design GPU Inference Serving System, alongside Google's ML System Design (Recsys / Chatbot / Image Classifier) and Apple's Siri Grounded Response Generation.

Our take: have a full-length serving design ready before the phone screen, not after it. If the phone screen is your first design conversation with the company, a shallow serving answer there ends the loop before the onsite where you planned to go deep.

What follow-ups should you rehearse for each design?

In our view, rehearse scale and failure follow-ups before new features. TrueInterview says its questions come with the follow-ups the interviewer asks, and the Anthropic title Review and Scale an Inference Service Design puts the scaling follow-up in the prompt itself.

No claim here reports the follow-ups from a GenAI round, so the clearest example of the style comes from outside GenAI. In one candidate's reported rate limiter round, described on TrueInterview's site, the interviewer asked what happens across three app servers and had the candidate redo the design with a shared store. TrueInterview lists a follow-up for that round on keeping the design correct under clock skew, and another that asks, "What breaks first at 10× the traffic?"

Apply the same questions to each GenAI design. For the inference server: what happens when a GPU fails mid-generation, how a client resumes a broken stream without paying twice, and which resource runs out first under heavy growth, usually KV-cache memory before compute. For the GPU scheduler: what happens when a node fails during a multi-GPU training job, and how the scheduler rebuilds its own state after a restart. For the chatbot: what happens when the retrieval index is stale or the primary model is unavailable, and whether you degrade to a smaller model or fail clearly.

Third-party guides describe a broader checklist. The System Design Handbook guide says interviewers check whether you understand that LLMs process tokens and how context window limits affect the design, test whether you cache to avoid redundant generation, and expect you to discuss routing simple queries to cheaper models or using quantization and distillation. The guide warns, "If you don’t bring up safety, it may be noted as an omission." Treat that as one guide's framework, not as the rubric OpenAI or Anthropic interviewers use.

What changes at senior and staff level, and for ML versus software engineers?

This topic is met mostly at senior level. Write-ups link these questions 89 times from candidates who reported a senior, staff-plus or manager level and 13 times from those who reported a junior, new-grad or intern level. The leading titles show the same senior-heavy split, so the prompt you draw says little about your level; in our view, depth is what separates levels.

The leading titles sit close to the overall split: Design GPU Inference Serving System has 24 senior-or-above links against 2 junior, Design GPU Scheduling Platform 23 against 5, and Design AI Prompt Playground 12 against 2. Staff-plus or manager write-ups link Design GPU Inference Serving System, Design GPU Scheduling Platform and Distributed Training Data Pipeline (FAR), one each. Levels are self-reported by candidates, and one staff-plus link per title is too few to describe a staff pattern.

Our take: at senior and staff level, expect the interviewer to assume the base architecture early and move the discussion to capacity, GPU utilization, failure handling and critiquing an existing design. Practise compressing the first diagram so most of the round goes to those trade-offs, and state the reason behind each decision before you are asked for it.

By role, software engineers account for 68 of the links, machine learning engineers 15 and AI engineers 12. Design GPU Scheduling Platform is linked from 25 software-engineer write-ups and 3 from machine-learning, AI, research or data-science candidates, while Design GPU Inference Serving System has 19 and 7. Two titles came only from machine-learning, AI or research candidates: Review and Scale an Inference Service Design with 4 links and Siri Grounded Response Generation with 2. Google's ML System Design (Recsys / Chatbot / Image Classifier) leans the same way, with 6 such links against 1 from a software engineer.

The infrastructure prompts are not reserved for ML engineers: software engineers supply most of the GPU scheduling links. If you apply as an ML or AI engineer, add the model-facing titles to your list, and remember that their counts are small.

Down-leveling note (our view): no claim here measures how a design round affects leveling. In our view, this round carries the level signal through depth. A candidate who draws a correct serving diagram but cannot reason about memory per request, utilization or failure recovery reads as mid-level. If you target senior or staff, prepare the follow-ups above as carefully as the base design, and use the project deep dive to show a system you scaled or operated.

How we counted

Counts come from TrueInterview's question bank as of 2026-11-02 and cover GenAI and AI infrastructure design questions filed under big-tech companies. A question can be filed under several companies and reported in several rounds, so tallies overlap. Write-up links count candidate reports, split by the level and role each candidate gave, and say nothing about how often a company asks a question. Problem families are assigned from question titles.

FAQ

How is a GenAI system design interview different from a traditional one?

The System Design Handbook guide says traditional system design interviews emphasize load balancing and data consistency. In the reported prompts, the bigger difference is the scarce resource. GPU memory and GPU time are what you schedule, so capacity reasoning starts from KV-cache size and batch size rather than from requests per second on stateless servers. A candidate who sizes a fleet of web servers well can still miss that a single long request can crowd out a batch.

What do interviewers evaluate in a GenAI design round?

None of the sources checked here publishes a company rubric. The System Design Handbook guide says interviewers typically evaluate candidates across five dimensions, and expects a composable pipeline of embedding models, vector databases, prompt composers, LLM gateways and output filters. Treat that as one guide's view. The most-linked reported prompts point to narrower tests by company. At Anthropic they centre on the serving layer; at OpenAI and Databricks the most-linked titles shown here pair GPU scheduling with a chatbot application design, though OpenAI files further titles not named in this article.

Do I need to know vLLM's internals?

No. Know the mechanism and its trade-off: block-based KV-cache allocation lets more requests share a GPU, which raises batch size and utilization, as explained in the serving section. vLLM's own documentation marks its PagedAttention design page as historical, so describing the paper's details as how vLLM works today is an error an interviewer can catch. Spend the time on admission control, queueing and preemption instead, which apply whatever framework the team uses.

Should I prepare RAG and chatbots or GPU infrastructure first?

Both, in an order set by your target. If your loop is at Anthropic, spend most of your design time on inference serving and batching, then the prompt playground. At OpenAI or Databricks, put GPU scheduling first and the chatbot app second. If it is at Apple, Microsoft, Amazon or Pinterest, the one dated example shown for each is a chatbot, memory or grounded-generation design, so check that company's full list in the practice bank and, if it matches, shift weight there while keeping one serving design ready. If you apply as an ML or AI engineer, add the model-facing titles from the role split above.

Are these the questions OpenAI and Anthropic officially ask?

No. They are candidate-reported prompts reconstructed for practice, as the note under the first table says. Use them to decide what to practise first. A title linked from many write-ups tells you candidates keep reporting it, not that a company draws from a fixed list, so prepare the problem family behind each title rather than memorising a single answer.

Your practice plan

A three-week plan for someone working full time, ordered by what candidates report:

1. First week, first half: build Design GPU Inference Serving System end to end: the request path, batching on each GPU, KV-cache memory per request, streaming and admission control. Write down what fails first when traffic grows tenfold.
2. First week, second half: drill Route and Batch Inference with Eight GPUs as a capacity exercise, then critique your own serving design with Review and Scale an Inference Service Design.
3. Second week: for an OpenAI or Databricks loop, build Design GPU Scheduling Platform with all-or-nothing placement, preemption, team quotas and checkpoint recovery. For an Anthropic loop, build Design AI Prompt Playground instead, with prompt versions, run history and per-user cost caps. If your target is NVIDIA or Oracle, add Telemetry Collector and GPU Utilization Dashboard or GPU Health and Repair.
4. Third week, first half: build one application design, Design AI Chatbot App or Chatbot Personalization / Memory System Design. Start from a single model call with retrieval and memory, then add routing, a guardrail and permission-filtered retrieval only as the requirements demand.
5. Third week, by target: for an Anthropic loop, add the GPU scheduler if time remains; for Apple or an ML role, add Siri Grounded Response Generation.
6. Final mock: run a timed mock of your serving or scheduling design with a partner who interrupts with scale and failure follow-ups. Before you call it ready, check that you covered memory per request, batching and GPU utilization, queueing and preemption, time to first token and streaming, retrieval quality and access control, and a hallucination check.

Sources

• TrueInterview question bank — GenAI and AI infrastructure design at big-tech companies — counted 2026-11-02
• PagedAttention · Hugging Face — checked 2026-11-02
• Paged Attention - vLLM — checked 2026-11-02
• Building effective AI agents \ Anthropic — checked 2026-11-02
• Chunking Strategies for LLM Applications | Pinecone — checked 2026-11-02
• Real FAANG Interview Questions by Company · TrueInterview — checked 2026-11-02
• Interview prep compared: your application, end to end · TrueInterview — checked 2026-11-02
• Generative AI System Design Interview: 2026 Guide — checked 2026-11-02

Last reviewed: 2026-11-02.