Compute CIs, power, and multiple testing

The baseline conversion rate is $p_0 = 0.051$ with $N_0 = 100{,}000$ users. You want to detect a +5% relative lift ($p_1 = 1.05 \times p_0$) at $\alpha = 0.05$ (two-sided) with 80% power.

Tasks:

1. Calculate a 95% Wald interval for $p_0$, and also an Agresti–Coull or Wilson interval. Explain why the Wilson or Agresti–Coull approach may be preferable.
2. Estimate the per-variant sample size needed for 80% power to detect the target lift, using a normal approximation for two proportions. Give all formulas and assumptions.
3. If you test three metrics at the same time (conversion, AOV, retention), apply a Bonferroni or Holm correction and give the adjusted $\alpha$ for each. Discuss the trade-offs relative to FDR control with Benjamini–Hochberg.
4. The data have user-level clustering because some visitors are repeat users. Explain why this breaks independence, how to correct the standard errors (for example, cluster-robust standard errors or user-level aggregation), and how that correction affects power.

Overview: This question assesses a candidate's ability in statistical inference for binary outcomes, confidence interval methods, power and sample size estimation, multiple-testing corrections, and handling clustered user-level data.