Decide best email variant using stratified A/B analysis

You have run an email A/B test across two strata defined by week and location.

Week 1 (Los Angeles): Variant A was sent to 100,000 recipients and produced 10,000 responses; variant B was sent to 10,000 recipients and produced 1,500 responses.

Week 2 (New York): Variant A was sent to 10,000 recipients and produced 400 responses; variant B was sent to 100,000 recipients and produced 6,000 responses.

Determine which variant is better.

Requirements:

1. Compute stratum-specific conversion rates and 95% confidence intervals.
2. Test for a common treatment effect across strata using a Mantel–Haenszel estimate (report the common odds ratio and 95% CI) and state the two-sided p-value ($\alpha = 0.05$).
3. Test for effect heterogeneity (e.g., Breslow–Day or an equivalent interaction test) and interpret the result.
4. Compute the naive pooled difference ignoring stratification, explain whether Simpson’s paradox occurs here, and why.
5. Provide a recommendation for either A or B, with a justification that reconciles the stratified and naive perspectives. List any assumptions you make clearly, such as independence or single exposure per user.

Overview: This question assesses skills in statistical inference and experimentation, specifically stratified A/B testing, estimating and interpreting treatment effects such as conversion rates, odds ratios, and confidence intervals, and evaluating effect heterogeneity across strata.