Uber · Statistics & Data Analysis
Explain and validate A/B test assumptions
TrueInterview
October 7, 2026 · 1 min read
Enumerate the core assumptions that must hold for a valid online A/B test, and for each one describe: (1) the formal meaning of the assumption; (2) a realistic product situation in which it breaks down; (3) a diagnostic you would apply to detect that breakdown; and (4) a concrete mitigation or redesign. Include at minimum: randomization integrity and sample ratio mismatch (SRM); independence/SUTVA and interference (such as network effects or shared inventory); stable unit exposure and cross-over/noncompliance; stationarity/time trends and novelty/learning effects; metric logging bias/missingness (MCAR/MAR/MNAR); sequential peeking and error inflation; and heterogeneous treatment effects across key segments. For every assumption, state precisely how you would carry out the check—for example, which statistical test or visualization, which pre-experiment covariates, how long a pre-period, and whether to adopt cluster randomization, stratification, CUPED, or staggered ramps.
Overview: The question assesses skill in experimental design, causal inference, and statistical diagnostics for A/B testing, spanning areas such as randomization integrity, SUTVA/interference, noncompliance, time-varying effects, metric missingness, sequential testing, and heterogeneous treatment effects.