Airbnb · Statistics & Data Analysis
Estimate impact of global launch without holdout
TrueInterview
October 7, 2026 · 1 min read
On 2025-05-10 a product feature rolled out worldwide, with neither a control group nor a holdout in place. Build a plan to estimate the causal lift it produced in weekly revenue and in 7-day retention. Put forward at least two identification strategies that stand independently of one another (for example: an interrupted time series adjusted for covariates, synthetic control assembled across markets, difference-in-differences against unaffected cohorts, regression discontinuity at the launch timestamp, or an IV/front-door design driven by eligibility or latency shocks). For each strategy, spell out: (a) the identification assumptions and the threats to them; (b) the unit of analysis, the data required, and how long the pre-period must run; (c) diagnostics and falsification tests (placebo dates, negative and positive control outcomes, pre-trend checks); (d) how uncertainty is quantified and 95% CIs produced (delta method versus bootstrap, and the clustering level); (e) how effects are bounded when assumptions hold only partially (e.g., amplification/robustness curves, sensitivity to unobserved confounding); (f) how you handle interference and spillovers, seasonality and holidays, concurrent marketing, logging or schema changes, backfilled events, selection into exposure, and migration; (g) a decision framework for reconciling conflicting estimates and arriving at a single recommendation — ship, rollback, or iterate — including acceptable risk thresholds.
Overview: This question tests a candidate's competence in causal inference, observational study design, and statistical uncertainty quantification for product analytics, specifically measuring lift after a worldwide launch with no holdout while contending with confounding, interference, and operational data-quality issues.