What If You Can't Randomize Users in an A/B Test Interview?

Keep randomization and move it: assign whole clusters such as regions, or switch a whole market between treatment and control over time, then prove the split is trustworthy with guardrails and a sample ratio mismatch (SRM) check. As of 2026-11-02, TrueInterview's bank holds 29 candidate-reported experiment design questions from 13 big-tech companies. Across that whole family, candidates reported 21 phone-screen and 8 onsite rounds, a split for experiment design questions overall rather than for interference prompts alone. Our take: build a compact answer you can give early in a phone screen. Name the interference path first, pick the unit, state what that unit costs you and close with the health checks.

Disclosure: TrueInterview is an interview-preparation product and publishes this article. Facts about other products come from their public pages on the dates listed under Sources.

Where does this prompt show up in a data science loop?

Experiment design questions as a family show up mostly in phone screens, while the marketplace interference prompts were reported in both rounds. The reported round tags for all 29 experiment design questions are 21 phone-screen and 8 onsite, and one question can carry more than one tag. Of the 21 phone-screen questions, 19 sit in the A/B testing and experiment design family. That split describes the whole family, not the narrower prompts where users interfere with each other, so have a compact design answer ready early in the loop.

Of the marketplace interference prompts, Uber's Design a switchback and choose block length (last reported 2026-01), Airbnb's network-aware Wi-Fi badge prompt and Instacart's pricing prompt with network effects came from phone screens, and DoorDash's marketplace product changes prompt from an onsite. Our reading: four reports are too few to show which round this prompt usually lands in, so prepare it for both.

By company, Meta accounts for 9 of the questions, ByteDance, Airbnb and Uber for 3 each, DoorDash and Google for 2 each, and OpenAI, Pinterest, Instacart and Amazon for 1 each. Meta has the most reported questions, and the two Meta prompts this article names are a feed ads test with guardrails and an experiment on fake-account impact. The marketplace companies, Uber, Airbnb, DoorDash and Instacart, supply the prompts where interference between buyers and sellers is the whole problem.

The target is narrow. Of the 29 questions, 23 fall in the A/B testing and experiment design family and 3 in the ads, recommendation and ranking family. Seven were last reported in the 12 months before 2026-11-02. One recent example is OpenAI's Design a free-month experiment (phone screen, last reported 2026-02).

These are candidate-reported questions reconstructed for practice, not any company's official question list. Treat the counts as a map of what candidates saw, then spend your first week on a short design answer you can deliver early in a phone screen. Layer the onsite-depth trade-offs on top only after that answer is automatic.

Can you randomize by user?

Only when treatment and control users share nothing the treatment changes. Statsig's SRM guide describes random assignment of individuals to control or test as one of the fundamental assumptions of a randomized experiment. Interference leaves that assignment intact but breaks what it is supposed to buy you: one user's outcome now depends on other users' assignments, so the comparison no longer isolates the treatment.

Statsig's marketplace guide says traditional per-user randomization creates cross-group contamination, where "one side of the marketplace in treatment interacts with another side in control". The same Statsig guide warns that even a simple fee change can push demand or supply so that treatment "steals" traffic from control, inflating or deflating the metrics.

The bank has prompts built on exactly this problem. Instacart's Design a pricing experiment with network effects was reported in a phone screen. DoorDash's Design experiments for marketplace product changes was reported in an onsite, last in 2026-02.

Our take: lead with the mechanism, not a menu of methods. Make your first sentence follow one template: treatment and control share X, so the per-user estimate is biased in direction Y. For a price cut on one side of a delivery marketplace, the shared X is local courier or shopper supply, and the bias usually overstates the gain, because treated customers take capacity that control customers would have used. An interviewer who hears the mechanism first knows you will pick a unit that cuts it.

What unit do you randomize instead?

Move randomization up a level instead of abandoning it. Statsig's marketplace guide says "randomized tests remain the gold standard for measuring causal impact" even with these hurdles. The two standard moves are clusters and switchbacks. Cluster randomization assigns whole cities, categories or geographic regions rather than individual users, according to Statsig's guide.

Switchback testing, as Statsig's guide describes it, turns a single unit such as a city or an entire platform into treatment or control at different times. Where network effects matter a lot, Statsig suggests sticking to clusters or switchbacks because they keep the traffic split stable.

Statsig's guide also warns that in a marketplace with strong network effects, bandits can cause problems if one variant starts pulling in users at the expense of the other. A bandit shifts traffic toward the apparent winner, and under interference that apparent win can be traffic taken from the loser.

Our take: pick the winner by where the spillover lives. If the spillover stays inside a geography or a category and the market splits into enough such units to balance them, use clusters. If the spillover runs through one shared pool inside a city, such as drivers or couriers, and the effect shows up within hours, use a switchback. For social features, cluster the graph so that most of a user's connections share an arm. Keep one sentence ready on why a bandit is the wrong tool here, because an interviewer may offer it as a shortcut.

What does each design cost you?

Each fix buys unbiasedness with power or time. Statsig's guide says clusters can differ in important ways, such as economic trends or population size, so you need enough clusters to balance those differences. The same guide says cluster designs typically need a larger sample size to get clear, reliable results. For staged rollouts, Statsig advises giving each phase enough time to stabilize, since supply and demand take a while to adjust.

Our take: the trade-offs are where the points are, so state the cost before the interviewer asks. For clusters, the effective sample is the number of clusters, not the number of users, so reduce variance by pairing or stratifying clusters on pre-period metrics and analyse at the cluster level or with clustered standard errors. For switchbacks, the cost is carryover and fewer independent periods. For a ramp, the cost is calendar time, and an early phase read before the market settles will mislead you.

Our default: a city-level pricing or dispatch change usually favours a switchback, because the market reacts fast and cities differ too much for a small set of clusters to balance. A catalogue or merchant-facing change favours category or region clusters, because the effect is slow and the spillover stays inside the cluster.

How long should a switchback block be?

Long enough for the market to reach its new state, short enough to give you enough blocks for power. Statsig's guide tells you to watch for "carryover" effects that might spill into the next time block. Uber's reported phone-screen prompt asks for exactly this decision: design a switchback and choose the block length.

Our take: defend the length from the system's own clock. Start from how long a change takes to work through the market, for example how long open orders or a dispatch queue take to clear, and set the block to several times that. Then drop a burn-in window at the start of each block so the previous arm's effects wash out, randomize the order of blocks, and balance arms across hours of the day and days of the week so that demand cycles do not line up with treatment.

State the tension out loud. Longer blocks cut carryover but leave fewer blocks and less power; shorter blocks give power but leak the previous arm into the next. A senior answer names the measurement that would settle it, such as comparing the first and second half of each block to see whether the effect is still moving.

What if you cannot randomize at all?

Say why, then fall back to a quasi-experiment with its assumption stated. A Statsig post says A/B tests "can't answer every question" and that other experiment designs can help avoid statistical pitfalls. The bank has prompts at this end of the spectrum: Amazon's Design causal study for reminder impact (phone screen) and Meta's Design experiment for fake accounts impact (onsite).

Our take: these sources do not cover observational methods, so the following is our guidance. Use difference-in-differences when a comparable group and a pre-period exist, and defend parallel trends with the pre-period plot. Use a synthetic control when only one market is treated. Use a regression discontinuity when a threshold, such as a risk score, decides who is treated. Whichever you choose, name the assumption that would break it and the check you would run: a pre-period trend plot for difference-in-differences, pre-period fit for a synthetic control, and a density check around the cutoff for a regression discontinuity.

What if exposure is not the same as assignment?

Report the effect of assignment first, then estimate the effect on those exposed. Statsig's SRM guide lists enrollment being decoupled from exposure as one cause of SRM to investigate. OpenAI's free-month prompt is a natural case for this follow-up, because a free month can be offered without being used (phone screen).

Our take: none of the verified sources cover noncompliance, so this is our own guidance. The intent-to-treat estimate compares everyone assigned to treatment with everyone assigned to control. It is unbiased for the policy you would actually ship, because you can only offer the feature, not force its use. If the interviewer wants the effect on people who used the feature, treat assignment as an instrument and scale the intent-to-treat effect by the difference in uptake between arms, which gives the effect on compliers. Say that this complier effect does not describe people who would never use the feature, and never compare users who chose to engage with users who did not.

Which health checks belong in the design?

An SRM test on the assignment unit and a short list of guardrail metrics, both decided before launch. Statsig's SRM guide puts it directly: "Sample ratio mismatch is evidence of selection bias". The same guide says SRM likely means some issue in the experiment's implementation. Google's reported prompt Design an A/B test with guardrails and SRM checks was a phone-screen question, last reported 2026-06.

That Google question is linked from 2 candidate interview write-ups. Meta's Design a feed ads A/B test with guardrails shows the same pattern in ads, alongside Pinterest's Decide if ad load is optimized. Our reading: when a reported prompt names guardrails, expect the checks to matter as much as the primary metric.

Name concrete causes when SRM fires. Statsig's SRM guide lists crash-rate differences when new code introduces or resolves crashes and data processing that deletes or duplicates data for only one variant. Statsig considers exposures unbalanced when the SRM p-value is below 0.01, which is Statsig's own rule, not a universal standard.

On the response, the author of Statsig's SRM guide says knowing the root cause is necessary before deciding whether to use the results. The guide calls re-running the experiment as designed, once the issue is found, the most scientifically rigorous approach. If results are used anyway, the same guide says controlling for pre-experiment bias with CUPED may help.

Our take: run SRM on the unit you randomized. In a cluster design, test the count of clusters per arm and the traffic within them; in a switchback, test the number and length of blocks per arm. A per-user SRM check on a cluster design can pass while the real assignment is broken.

When do you call the launch?

When the primary metric moves, no blocking guardrail breaks and the ramp has settled. ByteDance's Decide launch of downranking suspected bad sellers was reported in a phone screen, and Pinterest's Decide if ad load is optimized in an onsite. Both are launch decisions where one side of the platform pays for gains on the other.

Two statistical traps sit next to this decision. A Statsig post calls the multiple comparisons problem one of the most common pitfalls in A/B testing. Another Statsig post says sequential testing can support earlier decisions but is not right for every experiment.

Our take: write the launch rule before the test runs. Name one primary metric, the guardrails that block a launch on their own, such as seller retention for the ByteDance prompt or user engagement for the ad-load prompt, and the ramp steps. If you plan to look at the data early, say which sequential method you will use; otherwise commit to the planned duration. Then state the result that would make you say no, for example a significant drop in seller retention even with a gain in buyer conversion.

Which design fits which reported prompt?

The marketplace prompts call for clusters or switchbacks, and the feed and ads prompts call for guardrails and ranking-side interference. Uber's Design Pricing Model Experiment sits in the A/B testing and experiment design family next to the switchback question. Airbnb's network-aware Wi-Fi badge prompt was reported in a phone screen.

The table's company and round columns come from the reported prompts above; the last two columns are our view of the strongest answer.

Our take: let the company set your emphasis. If your loop is at a marketplace or delivery company such as Uber, DoorDash, Instacart or Airbnb, spend most of your preparation on clusters against switchbacks and on block length. If it is a feed or ads product such as Meta, Pinterest or ByteDance, spend it on guardrails and on interference between users in ranking.

Worked example: the Instacart pricing prompt

Take Instacart's Design a pricing experiment with network effects from the table above and answer it in four moves. This is our worked answer built from the guidance above, not a reported solution.

First, name the interference. Customers in both arms draw on the same local shoppers, so a price change in treatment can pull orders and capacity away from control and bias the measured effect, the fee-change problem Statsig's guide describes.

Second, pick the unit. Randomize whole regions as clusters, or alternate the price over time blocks inside a region as a switchback, the two designs Statsig's guide describes. Say what each costs: by Statsig's account, clusters need enough regions and more sample, and a switchback needs a block length that limits carryover.

Third, add health checks. Run SRM on regions or blocks, not users, and name the guardrails, such as shopper earnings and delivery lateness, that would block a launch. If SRM fires, find the root cause and re-run, the approach Statsig's SRM guide calls the most rigorous.

Fourth, plan the rollout. If you ship gradually, ramp by region and let each phase settle before you read it, because, as Statsig's guide notes, supply and demand take a while to adjust.

What changes at senior and staff level?

The bank has no level breakdown for this topic, so this section is our view. A senior or staff candidate chooses the design without being prompted, quantifies its cost, names the guardrails that block a launch and explains what they would do if SRM or interference undermines the result. A more junior answer that correctly describes one alternative design may be enough.

For background on how much large companies rely on experiments, Google's overlapping-experiment paper says the company evaluates "almost every change that potentially affects what our users experience". The same paper says experiment infrastructure alone is insufficient and discusses the tools and educational processes needed to use it well. Our view: a senior answer should say which results to distrust, such as a switchback read before the burn-in window or a cluster test with SRM on the cluster count.

Our take: at senior level, expect the interviewer to push on two decisions: the randomization unit and the launch rule. For the unit, explain why you rejected the per-user split and what the alternative costs in power. For the launch rule, say which guardrail would stop you and what you would do with an ambiguous result. Staff candidates should also say how they would make the design repeatable for other teams, for example a standard switchback template with an agreed block length and SRM check.

Down-leveling note (our view): in data science loops, the level signal sits in the depth of the design discussion and in the launch decision. A candidate who lists clusters and switchbacks but cannot say which to use, or who skips SRM and guardrails, gives the panel a reason to level down even when every statement is correct.

How we counted

The counts come from TrueInterview's question bank as of 2026-11-02: experiment design questions that candidates reported at big-tech companies, grouped into problem families from their titles. They show what candidates reported, filed by company, round and month, not how often any company asks a question, and the bank carries no level breakdown for this topic. Round tags can overlap because one question can be reported in more than one round.

FAQ

What is interference in an A/B test?

Interference means one unit's outcome depends on another unit's assignment, which breaks the stable-unit assumption behind a simple treatment-versus-control comparison. In a marketplace, treated buyers can use up supply that control buyers needed, so the gap between arms mixes the true effect with the spillover. In an interview, name the shared resource and the direction of the bias before you choose a design, because the shared resource tells you which unit to randomize.

How do you handle a user who is both a buyer and a seller?

Pick one identity for assignment and keep it for both roles, so the same person never sits in treatment as a seller and in control as a buyer. If the platform cannot link the two accounts, say so and treat it as a source of contamination that pushes you toward clusters. Our view: if this comes up, it tests whether you think about the unit of analysis before the metric, so state the identifier you would use.

How should you structure the answer in a phone screen?

Use a fixed order: the goal and primary metric, the interference path, the randomization unit and its cost, guardrails and the SRM check, then the launch rule. Deliver the whole skeleton first, one or two sentences per part, then ask whether the interviewer wants depth on the unit, the power or the launch rule. That keeps you from spending the call on power calculations before the interviewer has heard your design, and it lets the interviewer steer you toward the part they score.

Do you need the formulas for cluster or switchback power?

Explain the mechanism first and keep the algebra short. Say that power depends on the number of clusters or blocks, that similar units within a cluster add less information than independent users, and that pairing units on pre-period metrics reduces variance. If the interviewer asks for a formula, the design effect for clustered samples is the standard one to know, but a clear explanation of why the effective sample shrinks usually carries the point.

Your practice plan

A three-week plan for someone working full time, with one weekday session and one longer weekend session each week.

1. In the first week, write a one-page answer skeleton: interference path, randomization unit, cost of that unit, guardrails, SRM check and launch rule. Rehearse it aloud until you can deliver it early in a phone screen.
2. Also in the first week, work Design a pricing experiment with network effects end to end with the skeleton, stating clusters against switchbacks and picking one.
3. In the second week, work Design a switchback and choose block length and defend the block length from how fast the market reacts, including the burn-in window and how you would check for carryover.
4. Also in the second week, work Design an A/B test with guardrails and SRM checks and memorise the SRM causes and the response: find the root cause, then re-run.
5. In the third week, match your target company: marketplace candidates should work Design experiments for marketplace product changes; feed and ads candidates should work Decide launch of downranking suspected bad sellers.
6. Finish with Design causal study for reminder impact to practise the case where you cannot randomize, stating your method and the assumption that would break it.

Sources

• TrueInterview question bank — experiment design at big-tech companies — counted 2026-11-02
• A quick guide to sample ratio mismatch (SRM) — checked 2026-11-02
• Marketplace challenges in A/B testing and how to address them — checked 2026-11-02
• Statsig Blog | The authoritative source for data-driven insights — checked 2026-11-02
• Overlapping Experiment Infrastructure: More, Better, Faster Experimentation — checked 2026-11-02

Last reviewed: 2026-11-02.