Formulate hypotheses and compute AB test significance

Use the A/B test snapshot below for the pickup ETA card experiment to answer every part.

Data (7-day snapshot):

• Primary metric (trip completion rate per request): • Control A: $n_A = 50{,}000$ requests, $c_A = 6{,}000$ completions • Treatment B: $n_B = 50{,}000$ requests, $c_B = 6{,}420$ completions
• Guardrail 1 (rider cancel rate per request): • Control A: $cancels_A = 4{,}500$ • Treatment B: $cancels_B = 4{,}950$
• Guardrail 2 (wait time minutes, per request): • A: $mean_A = 4.8$, $sd_A = 3.2$, $n_A = 50{,}000$ • B: $mean_B = 4.7$, $sd_B = 3.4$, $n_B = 50{,}000$
• Five interim looks occurred at equally spaced information times, with no pre-registered alpha spending.

Tasks:

1. Write the exact $H_0$ and $H_1$ for the primary metric; say whether the test is one-sided or two-sided and explain why.
2. Pick the right test for the primary metric (difference in proportions), then compute the test statistic, p-value, and a 95% confidence interval for the lift. Show the formulas and the numeric results.
3. For Guardrail 2 (mean wait time), choose the correct test (for example, Welch’s t-test) and compute the 95% confidence interval for the mean difference. State any distributional assumptions and explain why Welch rather than pooled.
4. Apply a multiple-testing correction across the three outcomes (Primary, Guardrail 1, Guardrail 2) with Holm–Bonferroni at a familywise $\alpha = 0.05$. State which effects remain significant.
5. In plain language, explain what the p-value from part (2) does and does not mean.
6. Because there were 5 unplanned interim looks, reassess significance with a simple Pocock or O’Brien–Fleming alpha-spending method; outline the approach and give an approximate adjusted conclusion. Exact boundaries are not required, but justify your choice.
7. If the pre-period completion rate per rider has correlation $r = 0.40$ with the in-experiment outcome, estimate the approximate variance reduction from CUPED and discuss how that would affect required sample size or interpretation.
8. Conclude: ship, iterate, or stop? Defend your decision while taking the guardrails into account.

Overview: This question tests a data scientist’s ability in experimental design and statistical inference for A/B testing. It covers hypothesis formulation, difference-in-proportions testing and confidence intervals, guardrail analysis, multiple-testing correction, interim alpha-spending approaches, and variance-reduction techniques such as CUPED.

This item is drawn from a data scientist interview experience.