Interpret A/B results for video-pin increase

Pinterest is running a test that increases the share of video pins shown to new users in the home feed, aiming to lift engagement.

Among eligible new users, 50% are assigned to Treatment and 50% to Control.

Primary metric: 7-day time spent per user (minutes), tested one-sided (Treatment $>$ Control) with $MDE = +2\%$ relative to Control.

Secondary metrics: overall CTR (total clicks / total impressions) and D7 retention (users retained on day 7 / assigned users).

Observed after 14 days:

Assignments (users): Control 98,750; Treatment 101,250

Primary metric (per-user): Control mean 12.00, sd 8.00; Treatment mean 12.36, sd 8.00

CTR (aggregate): Control had 150,000 clicks out of 5,000,000 impressions; Treatment had 171,600 clicks out of 5,200,000 impressions

D7 retention: Control retained 21,725 of 98,750 users; Treatment retained 22,680 of 101,250 users

Answer the following:

1. Give the exact $H_0$ and $H_1$ for the primary metric, with the direction and the MDE.

2. Test for sample ratio mismatch (SRM) with a chi-square test at $\alpha = 0.001$. Given the observed assignment counts for a planned 50/50 split, should SRM be suspected?

3. For the primary metric, calculate the absolute lift, the relative lift as a percentage, a 95% confidence interval for the difference in means, and the one-sided $p$-value. Is the result statistically significant at $\alpha = 0.05$?

4. For CTR and D7 retention, perform suitable two-proportion tests and apply a Holm–Bonferroni correction for multiple comparisons across the two secondary metrics at a familywise $\alpha = 0.05$. Which metrics, if any, remain significant after adjustment?

5. Give a ship/no-ship recommendation. If you find SRM or other threats to validity (such as novelty effects, outliers, or country mix shifts), explain how they affect your decision and which additional diagnostics or guardrail checks you would run before launch.

Overview: This question assesses statistical inference and experimentation skills: formulating A/B test hypotheses with a predefined MDE, estimating absolute and relative lift with confidence intervals, testing proportions for CTR and retention, detecting sample ratio mismatch, and applying multiple-comparison adjustments.