ByteDance · Statistics & Data Analysis
Design robust A/B test with interference and seasonality
TrueInterview
October 7, 2026 · 1 min read
You are about to launch a revamped onboarding flow that is expected to lift Day-7 activation but may also create network effects (users inviting other users) and weekly seasonal patterns. Outline an experiment plan that covers: (1) the hypothesis, primary metric(s), guardrail metrics, and precise metric definitions including attribution windows; (2) the randomization and exposure unit (user, household, geo, or cluster) and the rationale, given possible interference; (3) sample size and power calculations, the target MDE, duration assumptions, and how you will handle seasonality (for example, using full-week multiples); (4) variance reduction approaches (such as CUPED with pre-period covariates), stratification, or geo-matched pairs; (5) a plan for detecting and remediating SRM; (6) a sequential monitoring method and stopping rules (alpha spending) to prevent p-hacking; (7) a ramp plan with holdout groups and a plan for novelty or learning effects; (8) checks for noncompliance, bot traffic, and triggered versus assigned populations; (9) how you would detect and reduce interference or spillovers (cluster randomization, geo experiments, or switchback designs) and quantify any bias if user-level randomization is chosen; (10) an interpretation plan when primary and guardrail metrics conflict, and how you would decide whether to ship. Overview: This question assesses expertise in experiment design, causal inference, statistical power and minimum detectable effect estimation, variance-reduction methods, sequential monitoring, and diagnostics for interference, spillovers, and weekly seasonality in A/B testing.