Compute power and interpret uplift metrics

A two-arm conversion experiment. The baseline conversion rate is $p_0 = 0.120$. The target is an absolute lift of $\Delta = +0.005$, with a two-sided $\alpha = 0.05$ and power of $0.80$. (1) Approximate the sample size needed per variant using the normal approximation for the difference between two proportions; include the formula and the computation. (2) After running for one week, the observed data are: control $n_C = 150{,}000$, $x_C = 18{,}000$; treatment $n_T = 150{,}000$, $x_T = 18{,}900$. Calculate the point estimate $\hat{\Delta}$, a 95% confidence interval, and a p-value; then discuss business significance versus statistical significance. (3) If CUPED is applied with a pre-period covariate that gives $R^2 = 0.30$ for the outcome, estimate the new effective sample size or variance and the updated MDE; show the work. (4) Four guardrail metrics are tracked with unadjusted p-values $\{0.03, 0.01, 0.20, 0.04\}$. Apply Holm–Bonferroni with familywise $\alpha = 0.05$ and identify which guardrails stay significant; show the ordering and the adjusted thresholds. (5) Describe how you would diagnose and adjust for overdispersion or miscalibration in conversion estimates when users are clustered by geography. Overview: This item assesses a data scientist's practical and conceptual skills in A/B test design and analysis under Statistics & Math, including power and sample-size calculations, binary outcome inference, CUPED variance reduction, multiple-testing corrections, and handling clustered or geographic data.