LLM Inference Interview Questions: KV Cache, Batching and GPUs

LLM inference interview questions at AI labs and big tech are mostly design and coding problems about GPU capacity, serving and resource accounting, not KV-cache trivia. As of 2026-10-16, TrueInterview's bank holds 29 candidate-reported inference and serving questions at 13 big-tech companies, and 14 of them are system design against only 5 knowledge questions. The two most-linked titles, Design GPU Scheduling Platform and Design GPU Inference Serving System, are both design prompts with 22 write-ups each. Learn KV cache, batching and quantization as inputs to a capacity and scheduling argument you can defend at a whiteboard.

Disclosure: TrueInterview is an interview-preparation product and publishes this article. Facts about other products come from their public pages on the dates listed under Sources.

Which LLM inference questions do candidates report at AI labs and big tech?

Candidates have reported 29 inference and serving questions across 13 companies, led by OpenAI with 11, NVIDIA with 5 and Anthropic with 4, with Uber, Snowflake, Databricks, Amazon and DoorDash at 2 each. Phone screens account for 17 of them and onsites for 12, with 3 in online assessments, and one question can show up in more than one round.

Candidates reported these questions in phone screens at least as often as in onsites, so start preparing before the recruiter books the loop. 24 of the 29 were last reported in the 12 months before 2026-10-16, which makes this a current set rather than an archive. The counts measure what candidates wrote up, not how often a company asks, and the questions are reconstructed for practice rather than taken from any company's own list.

The table ranks the named questions by candidate write-ups. Senior-or-above means the candidate self-reported a senior, staff-plus or manager level; ML-side means machine-learning, AI, research or data-science candidates; n/a means the bank has no tag for that cell.

Read the table as a prep budget. Our take: the top two rows are both design prompts, so if you can draw a serving system and a scheduler end to end, you have rehearsed the shape that leads the reports. The concept questions that fill generic lists appear here as 5 knowledge questions, 2 each filed under OpenAI and NVIDIA, plus a few ML-side drills.

How do you design a GPU inference serving system?

Treat Design GPU Inference Serving System as a memory and admission problem with a latency target, not as a generic load-balancer diagram. It is Anthropic's lead inference prompt, and all 4 inference questions filed under Anthropic are system design, so at Anthropic this is the format to rehearse first. Every write-up linking it came from a senior-or-above candidate (see the table).

The prompt sits in the bank's LLM, RAG and AI infrastructure family next to Route and Batch Inference with Eight GPUs, also filed under Anthropic. By its title, the second is the routing and batching core of the first, so practise them as a pair: one end-to-end design, one deep dive on placing requests across a fixed set of GPUs.

What to prepare for at senior level (our view): two decisions carry this design. The first is admission: how many sequences you let onto a GPU given weights plus KV state, and what happens to the next request when the GPU is full. The second is the batching and priority policy, which you should tie to tail time to first token rather than to average tokens per second. Practise the follow-up where the traffic mix changes halfway through, for example long prompts with short answers, and ask which of your limits moves.

Our take: start the whiteboard with the memory budget from the KV-cache section below, derive the concurrency limit, and only then draw the router, queue and workers.

How do you design a GPU scheduling platform?

Design GPU Scheduling Platform is the one inference prompt reported from three companies at once, OpenAI, Anthropic and Databricks, and the bank files it under Netflix as well. It is also the most junior-visible design prompt in the table, so a candidate at any level should expect it as a scheduling-and-fairness design rather than a model question.

The closest neighbour in the bank is GPU Health and Repair, a system design question a candidate reported in an Oracle onsite in 2026-09. Pair the two: the scheduler decides where jobs go, and the health system decides which GPUs are allowed to receive them.

What to prepare for at senior level (our view): practise two follow-ups. First, the queueing and preemption policy: priorities, per-team quotas, and what you do with a long training job when an inference workload needs the capacity. Second, fragmentation and placement: a job that needs eight GPUs on one host cannot run on eight scattered single GPUs, so explain how you reserve or pack capacity and what that costs in utilisation.

Our take: this prompt is won on the fairness rule. Name one policy, such as weighted fair share with a preemption budget, defend its failure mode, and give the metric that would tell you it is wrong. Avoid listing five possible policies without choosing one; it makes a strong engineer sound unsure.

How much KV cache fits on a GPU, and what does quantization change?

Start every serving answer with a memory budget: model weights plus per-sequence KV state, against GPU memory. The vLLM team writes that "the performance of LLM serving is bottlenecked by memory", and Anyscale's write-up says throughput is largely set by how large a batch fits into high-bandwidth GPU memory. The budget is what turns vocabulary into a concurrency number.

The definitions are short. During autoregressive decoding, the attention key and value tensors for earlier tokens are kept in GPU memory to generate the next token, and those cached tensors are the KV cache. vLLM's post puts the cache at up to 1.7GB for a single LLaMA-13B sequence and notes that its size depends on sequence length, which is highly variable and unpredictable. Anyscale's post adds that GPU memory use scales with base model size plus token sequence length.

The arithmetic we would put on the whiteboard (our view): per-token KV bytes equal layers times KV heads times head dimension times bytes per element, times two, then multiply by tokens per sequence and by active sequences. PracHub's guide explains that the factor of two is for keys and values, and that grouped-query or multi-query attention can reduce KV heads relative to query heads. Anyscale's back-of-the-envelope example lands at about 28 sequences per batch when sequence length is capped at 512. Practise producing that kind of number in under two minutes for a model you name.

Each lever then changes one term of the budget:

vLLM's post says PagedAttention takes its idea from virtual memory and paging in operating systems, and that shared blocks use reference counts with copy-on-write. Hugging Face's TGI documentation adds that cached KV blocks can be shared among generations of the same prompt in parallel sampling. Our recommendation: use the operating-systems analogy in the room, because it lets you explain fragmentation without a slide.

Our take: do not argue quantization on memory alone. PracHub's guide advises evaluating a representative quality suite rather than perplexity on one corpus. Aim to say what you would measure before shipping lower-precision weights and what result would make you roll back.

Continuous or static batching: which scheduling policy should you defend?

Defend continuous batching, then spend your time on the policy around it, because that is where design answers are won or lost. Static batching keeps the batch size fixed until inference completes, and Anyscale's post says that means "the GPU will be underutilized until the last sequence in the batch finishes generation". Anyscale also calls the alternative batching with iteration-level scheduling.

Naming the technique is table stakes. PracHub's guide lists the policy an interviewer expects you to state: maximum scheduled tokens, maximum active sequences, fairness or priority, cancellation, and what happens when the cache cannot admit another request. Pick a value or rule for each one and say why, because that list is the difference between a capacity plan and a buzzword.

Measure the policy the way a chat product feels it. PracHub's guide separates the request path into queueing, prefill and decode, and advises reporting p50 and p95/p99 rather than an average. PracHub's sharpest line is that "A service can have excellent total throughput and still fail a chat SLO if queued requests have high p99 TTFT."

What to prepare for at senior level (our view): prepare an answer for what you do when the KV cache is full and a high-priority request arrives. Have an answer that names the victim (queue it, preempt a low-priority sequence and recompute, or reject with a retry hint) and the metric that pays for the choice.

Our take: the vendor throughput multipliers in the vLLM and Anyscale posts are benchmarks under their own setups, so leave them out of your answer. Quote your own back-of-the-envelope number and your tail-latency target instead.

What do the inference coding questions actually test?

They test resource accounting and throughput under time pressure, and ordinary data-structure fluency transfers. Of the inference questions filed under OpenAI, 4 are coding, 3 are system design and 2 are knowledge, while the 2 filed under Uber and the 2 under Snowflake are all coding. By their titles, these read as accounting and throughput problems rather than GPU programming.

By algorithm topic, hashing and heap carry the strongest senior skew in these links, 10 senior-or-above against 2 junior-or-below each, while binary search runs the other way at 2 against 7. Across coding questions overall, senior-or-above candidates account for 22 links against 11 for junior-or-below. In practice, if you are senior, drill hash-map-plus-heap bookkeeping first; if you are earlier in your career, add binary search on the answer.

GPU Credits II and GPU Credit Calculator (OpenAI)

GPU Credits II is the most-linked inference coding question, reported in an OpenAI phone screen (see the table), and GPU Credit Calculator is its smaller sibling at OpenAI. By their titles, both are accounting problems over GPU usage.

What to prepare for (our view): the bank gives only the titles, so this is our reading of them: a ledger that supports grants, spends and expiry in timestamp order, built from a hash map per owner and a min-heap keyed by expiry. A natural extension to practise is out-of-order events or a query at an earlier time, and you should be ready to say what each operation costs.

Maximum Throughput (Uber, Snowflake) and Maximize Pipeline Throughput (Uber)

Maximum Throughput is linked from 8 write-ups at Uber and Snowflake, and only 2 came from senior-or-above candidates (see the table). Maximize Pipeline Throughput, a coding question a candidate reported in an Uber OA in 2026-06, has a single write-up, none of it from a senior-or-above candidate.

What to prepare for (our view): a sound default for throughput maximisation is binary search on the answer with a greedy feasibility check; have it ready and verify it fits the prompt. Time yourself: in an OA, the feasibility function is where the minutes go.

In-Memory KV Cache with WAL Log (Databricks)

This is a storage design problem, a key-value store with a write-ahead log, not the attention KV cache, and one of the most level-mixed titles in the table at 7 senior-or-above and 5 junior-or-below write-ups. It is listed here only because its title shares the KV-cache name. Do not let the shared name pull you into transformer internals.

What to prepare for (our view): append-before-apply ordering, replay on restart, and log compaction or snapshotting when the log grows. A senior answer also says what is lost on a crash between the write and the fsync.

How do ML engineers and software engineers get different questions?

Inference is not an ML-only topic: by the candidate's role on the write-up, software engineers account for 77 links and machine learning engineers for 17, with backend engineers at 5. The difference is which questions each group reports. Software engineers dominate the serving design, while three inference-internals titles have no software-engineer write-ups (see the table).

LLM Inference Timeout and Restart Strategy (OpenAI)

This OpenAI title has 4 write-ups, all from ML-side candidates. What to prepare for (our view): be ready to explain how you would detect a hung or slow generation, whether a restart resumes from cached state or recomputes the prompt, and how you would keep retries from pushing up tail latency for every other user in the queue.

GPU and Inference Systems Fundamentals (NVIDIA)

NVIDIA's inference questions split 2 system design, 1 coding and 2 knowledge, and this is the knowledge round, reported in a phone screen. What to prepare for (our view): why decoding is memory-bound, what the KV cache stores, and how quantization changes memory traffic, each explained in two sentences and then applied to a number. Anyscale's post notes that FlashAttention gained throughput by reorganising attention to need less memory-IO, which is a good example to have ready.

Debug a Transformer Implementation and Implement a KV Cache for Decoding

This is the one hands-on internals drill in the set, linked only from senior-or-above, ML-side write-ups (see the table). What to prepare for (our view): write the cache append and the attention call that reads it, then show that cached and uncached decoding produce the same logits on a short prompt. Shape bugs in the head and sequence dimensions are the usual place to look first.

ML-side candidates also reported Design a scalable recommendation serving system, a system design question from a DoorDash onsite in 2026-03, with no software-engineer write-ups. Amazon's Bucket Batching DP coding question, reported in an applied scientist phone screen in 2026-04, belongs on the same drill list.

Our take: if you are an ML engineer or researcher, add the KV-cache implementation and a fundamentals drill on top of the serving design, not instead of it. If you are a software engineer, weight the serving and scheduling designs far more heavily, since software engineers supplied most of the role-tagged links to the serving design.

What changes at senior and staff level?

Senior-or-above candidates account for 85 of the level-tagged write-up links to these questions, against 23 from junior, new-grad or intern candidates, and the skew is sharpest in system design at 57 against 8. Three titles appear only in senior-or-above write-ups: the serving design, the transformer debug and KV-cache drill, and Route and Batch Inference with Eight GPUs.

These levels are what candidates reported about themselves, so treat them as a direction rather than a rule. The direction is clear enough to act on: at senior and staff level, the inference questions you will meet are mostly design rounds.

Our view of the bar: at mid level, a clear explanation of PagedAttention and continuous batching is usually the core of what gets probed. At senior level, expect to defend a capacity number, a scheduling policy and the tail-latency trade-off you accepted. At staff level, add the operating model: how the fleet is shared across teams, how you roll out a quantized model safely, and what you would build first with half the GPUs you asked for.

Down-leveling note (our view): the level signal in these loops sits in the design depth of the serving or scheduling round. In our view, a clean coding round paired with a design answer that never produces a memory budget or a priority rule is the profile most at risk of an offer one level down.

Where do candidates lose inference rounds?

In our view, the most avoidable failure is preparing for the wrong format. Knowledge questions are 5 of the 29, far fewer than the 14 system design questions, so a memorised top-100 list prepares you for the minority of what candidates see. The other failures are specific and avoidable:

• Naming continuous batching with no limits. Give a max-sequences number and an admission rule, the policy items PracHub's guide lists, so the interviewer can judge your capacity plan.
• Optimising the average. PracHub's guide advises reporting tail percentiles alongside p50; design for the tail time to first token a chat user actually waits on.
• Quantizing on faith. State the quality suite and the rollback trigger before you claim the memory win.
• Confusing the two KV caches. The Databricks storage problem and the attention cache share a name and nothing else.
• Quoting vendor speedups. Use your own estimate; a multiplier from a blog post invites the question of which setup it came from.

PracHub's guide frames the same point from the concept side: a good answer connects quantization, batching, KV-cache capacity and latency metrics rather than claiming one technique speeds up every model.

Which inference questions should you prepare for, by company?

Prepare by the company's reported format mix, since it differs sharply between labs. Anthropic's 4 and Databricks' 2 inference questions are all system design, OpenAI mixes design, coding and knowledge, and Uber and Snowflake report only coding. Spend roughly two-thirds of your time on the format your target reports most.

Worked example: a senior software engineer has an Anthropic phone screen and an OpenAI loop on the calendar. For Anthropic, the serving design is the target, reported in a phone screen in 2026-07 and linked only by senior-or-above candidates. In our view, they should rehearse it with a memory budget first, continuous batching second and an admission and priority rule third, measured on tail time to first token. For OpenAI, they add the scheduling design and GPU Credits II, the coding question with 10 senior-or-above write-ups. Two designs and one coding drill cover the most-linked rows for both companies.

Our take: prepare for a design round first, a definitions quiz second

Concept lists treat inference as a knowledge interview. The KalyanKS LLM interview hub on GitHub lists questions such as "Explain how KV Cache accelerates LLM inference." and how quantization affects speed and memory. PracHub's guide is closer to our view: it says these interviews ask whether you can identify the bottleneck, choose a lever, predict the trade-off and prove it with a benchmark.

We agree with that framing and would add the format. In the reports, the concepts arrive inside a serving or scheduling design and a handful of accounting problems. Learn each concept until you can use it to change a number on the whiteboard, then stop: an hour on positional embeddings is an hour not spent drawing the serving system Anthropic candidates keep reporting.

How we counted

We counted the 29 questions in TrueInterview's bank tagged LLM inference and serving as of 2026-10-16. The tag also covers adjacent serving, caching and GPU-accounting problems, such as the Databricks key-value store and the credit questions. We took each question's company, format, round and last-reported month from the candidate write-ups that link to it. Level and role splits use the level and role the candidate self-reported on each write-up; links without a level or role are left out of those splits. A question can be filed under several companies and rounds, so totals across columns do not add up to the question count.

FAQ

What are the most common LLM inference interview questions?

In candidate reports, the most-linked are two design prompts, Design GPU Scheduling Platform and Design GPU Inference Serving System, followed by a Databricks KV-store problem and OpenAI's GPU Credits II coding question. The ranked table above gives the full list with companies, rounds and levels. Expect KV cache and batching to come up inside those designs rather than as standalone definitions, so practise the designs and bring the concepts with you.

How do you improve LLM inference in an interview answer?

Start from the bottleneck, which for serving is usually memory, then pick the lever that changes the term of the budget you are short on. vLLM's post says paging the KV cache lets a system batch more sequences and raise GPU utilisation. Then name the measurement that would prove the change on tail latency, and the result that would make you roll it back.

Are there LLM coding interview questions about inference?

Yes, and they are mostly resource-accounting and throughput problems rather than model code. OpenAI's GPU Credits II and GPU Credit Calculator and the Maximum Throughput problem at Uber and Snowflake are the reported examples, and by their titles they read as bookkeeping problems rather than GPU programming. The exception is the hands-on transformer debug and KV-cache drill, which ML-side candidates reported.

Do junior candidates get LLM inference questions?

Some do, but far fewer: junior, new-grad and intern candidates account for a small share of the level-tagged links, and none of them linked the Anthropic serving design. The titles with the most junior-or-below write-ups include the GPU scheduling design, the Databricks KV-store problem and GPU Credits II (see the table), and binary search is the algorithm topic that skews junior. If you are early-career, prepare those first and treat the serving design as a stretch goal.

Your practice plan

Here is a plan in weekly blocks for someone working full time.

1. Opening week: write a one-page memory budget for a model you know: weights, per-sequence KV state, sequences that fit, and what lower-precision weights or paged allocation change. Check it against the arithmetic in the KV-cache section above.
2. Opening week: solve GPU Credits II and Maximum Throughput under a timer, then rewrite each with cleaner bookkeeping.
3. Middle week: design GPU Inference Serving System out loud in one interview-length session, starting with numbers and ending with the admission rule and the tail-latency metric you would watch.
4. Middle week: design the GPU Scheduling Platform and pick one fairness policy you can defend, then sketch GPU Health and Repair as its companion.
5. Final week: if you are an ML engineer or researcher, implement the cache in Debug a Transformer and Implement a KV Cache and work through LLM Inference Timeout and Restart Strategy; if you are a software engineer, do Route and Batch Inference with Eight GPUs instead.
6. Final week: run a full mock on your target company's top row from the company table, and grade yourself on whether you state a capacity number, a scheduling policy and a tail-latency trade-off.

Sources

• TrueInterview question bank — LLM inference and serving at big-tech companies — counted 2026-10-16
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blog — checked 2026-10-16
• Achieve 23x LLM Inference Throughput & Reduce p50 Latency — checked 2026-10-16
• LLM Inference Optimization Interview Questions: Quantization, Batching, KV Cache, and Latency | PracHub Knowledge Hub — checked 2026-10-16
• PagedAttention · Hugging Face — checked 2026-10-16
• GitHub - KalyanKS-NLP/LLM-Interview-Questions-and-Answers-Hub: 100+ LLM interview questions with answers. · GitHub — checked 2026-10-16

Last reviewed: 2026-10-16.