Act when A/B result is not significant

An A/B test for the 60-second video change produces a non-significant lift on the primary metric at $\alpha = 0.05$. The baseline completion rate is 22%. The product's minimum meaningful effect is +5% relative (to 23.1%). Assuming 80% power, a two-sided test, and equal allocation: (1) calculate the required per-arm sample size and explain the formula and assumptions you used; (2) suggest a variance-reduction approach (such as CUPED with pre-period completion, or covariate adjustment) and estimate the power gain when $R^2 = 0.25$; (3) decide whether to keep collecting data, stop, or switch to a Bayesian decision with a utility-based threshold; (4) discuss how to control multiplicity if several related outcomes are monitored; and (5) outline a sequential design (for example, O’Brien–Fleming) that would preserve Type I error while permitting early looks.

Overview: This question tests a candidate's skills in experimental design and statistical decision-making, including sample size calculation, variance-reduction techniques, interpretation of non-significant A/B results, multiplicity control, and sequential/Bayesian decision frameworks.