Differentiate and control Type I/II errors

Imagine you are setting up a two-arm A/B test that measures sign-up conversion. The baseline conversion rate is $p_0 = 0.10$. The goal is to detect an absolute increase of $+0.01$ (from $0.10$ to $0.11$) with a two-sided significance level $\alpha = 0.05$ and power $1-\beta = 0.80$, using a two-proportion z-test.

1. Work out the necessary sample size per arm, showing the formula and the values you plug in.
2. Suppose you also track 10 secondary metrics and need to control the family-wise error rate using Bonferroni correction. What per-metric $\alpha$ should you use, and how does that affect the required sample size?
3. If you intend to check the results daily for 14 days, describe a valid sequential testing approach (such as O’Brien–Fleming or Pocock alpha-spending) and explain how it changes the stopping boundary.
4. Explain the practical consequence of a Type II error in this test, and give one concrete way to lower $\beta$ without raising $\alpha$, quantifying the trade-off.

Overview: This question tests knowledge of core hypothesis testing concepts: Type I and Type II errors, statistical power and sample size formulas for two-proportion tests, family-wise error rate control (e.g., Bonferroni), and sequential analysis with alpha-spending.

This question was shared as part of a data scientist interview experience.