Quantify launch decision with tests and guardrails

You are asked to formalize the statistical decision rules for the Instagram button experiment described above.

Assume a baseline exploration rate of $p_0 = 0.15$ per user-week, a desired absolute MDE of $+1.5$ percentage points, a two-sided $\alpha = 0.05$, and power of $0.80$. Randomization happens at the cluster level, with average cluster size $m = 500$ users and intra-cluster correlation $\text{ICC} = 0.02$.

(a) Compute the required number of users and clusters per arm from a proportions test adjusted for clustering, using the design effect $DE = 1 + (m - 1)\cdot \text{ICC}$. Show the formulas and the resulting final counts. (b) State the exact hypothesis test you would use for the primary metric, such as a cluster-robust z-test on cluster means or a user-level test with cluster-robust standard errors. Explain when a nonparametric alternative would be preferable. (c) Define the confidence interval you will report and how you will interpret it jointly with practical significance. (d) You will review the primary metric each week for four weeks. Choose and justify a sequential testing plan, such as O’Brien–Fleming alpha-spending, and give the adjusted per-look alphas. (e) You track three guardrails: p95 latency, crash rate, and add-to-cart rate. Describe a multiple-testing control that preserves power on the primary metric, such as hierarchical testing or Holm–Bonferroni, and write the decision logic that combines the primary metric with the guardrails. (f) Suppose contamination causes 10% of control users to see the button. State the direction of the resulting ITT bias and outline a correction, such as CACE with assignment as an instrument.

Overview: The question assesses a data scientist’s competence in experimental design, statistical inference, and causal-effect estimation for clustered A/B tests. It spans sample size calculation with a design effect, choice of cluster-robust hypothesis tests, sequential alpha-spending, multiple-testing control for guardrails, and quantifying contamination bias. It is commonly asked in the Statistics & Math domain to evaluate the ability to formalize decision rules that control type I and type II error rates and to interpret confidence intervals under clustering and interim looks, covering both conceptual understanding of statistical principles and practical experiment governance.

Read the full data scientist interview experience this question came from.

Community answers

Answer by SS

(a) Given: Baseline rate: $p_0 = 0.15$. Absolute MDE: $\delta = 0.015$, so $p_1 = 0.165$. Significance: $\alpha = 0.05$ two-sided, giving $z_{1-\alpha/2} = 1.96$. Power: $0.80$, giving $z_{1-\beta} = 0.84$. Cluster size: $m = 500$. ICC: $\rho = 0.02$.

Step 1: Sample size ignoring clustering Variance = $0.2653$. MDE = $0.015$. Users per arm = $7.86 \times 0.2653 / 0.015^2$. Users per arm without clustering: about $9,244$.

Step 2: Adjust for clustering with the design effect $DE = 1 + (500 - 1) \times 0.02$ $= 1 + 499 \times 0.02$ $= 1 + 9.98$ $= 10.98$

Step 3: Effective sample size with clustering Users with clustering = $10.98 \times 9244$. Adjusted users per arm: about $101,500$.

Step 4: Convert users to clusters $101,500 / 500 = 203$.

Final answer Per arm: Users required: about $101,500$. Clusters required: about $203$. Total experiment: Users: about $203,000$. Clusters: about $406$.

Key insight The roughly $11\times$ design effect sharply increases the required sample size because of clustering. Even a modest ICC of $0.02$ becomes expensive when the cluster size is as large as $500$.