Design and analyze A/B test with interference

You need to launch a News Feed ranking change in which content created by treated users may be shown to control users, producing interference and correlation within a user across sessions. Only logged-in traffic qualifies. The constraints are: a maximum 10% concurrent treatment ramp; a 14-day analysis window; sessions per user as the primary metric; guardrail metrics include crash rate and time spent; and policy requires sequential looks every 2 days.

Design and analysis questions:

1. Select the experimental unit and randomization approach that reduces interference (for example, graph or ego-network clustering, producer-side randomization, or two-stage randomized encouragement). Justify the choice under the 10% cap and a heavily skewed degree distribution.
2. Define the exact estimands: the direct effect on consumers, the spillover effect, and the total effect. State the exposure conditions and an exposure model that makes these estimands identifiable.
3. Describe the estimator and its variance: show how to compute point estimates and 95% confidence intervals using cluster-robust or randomization-inference standard errors. State the assumptions explicitly and explain how violations alter interpretation.
4. Quantify the design effect and required sample size under clustering: derive the required $N$ given an average cluster size $m = 120$ and ICC $\rho = 0.02$; show $n_{eff} = \frac{n}{1 + (m-1)\rho}$ and solve for $n$ to achieve 80% power to detect a 0.6% relative lift when the baseline mean is 4.0 sessions/user with SD 5.5.
5. Diagnostics: propose at least three concrete checks for detecting leakage or interference (such as cut-edge exposure rates, the share of treated-producer content seen by control users, and placebo effects among non-exposed control users), and describe how you would respond to each.
6. Sequential testing: choose a method (for example, alpha-spending with O’Brien–Fleming boundaries or always-valid tests) and outline stopping or decision rules that control Type I error across the interim looks.
7. If leadership requires user-level 50/50 randomization with no clustering, propose post-hoc adjustments to bound the bias and obtain conservative confidence intervals (for example, exposure-weighted IV, CUPED with a pre-period, or sensitivity bounds).

Overview: This question assesses skill in experimental design and causal inference for online A/B testing in the presence of interference, including competencies such as defining estimands and exposure models, handling clustered or dependent data, variance estimation and confidence intervals, power and sample-size calculations, sequential testing, and leakage diagnostics. It is frequently asked in Analytics & Experimentation interviews because it tests practical application of statistical design and analysis under production constraints, examines understanding of bias caused by cross-group interference and operational limits, and sits in the Analytics & Experimentation domain with a primary focus on practical application supported by conceptual understanding.