Analyze an A/B test over last 7 days

Suppose today is 2025-09-01. Over the previous 7 days (2025-08-26 through 2025-09-01), you ran a 50/50 A/B test on a checkout flow. The daily exposure and purchase counts are shown below:

Tasks:

1. Calculate the overall conversion rates for A and B, the absolute and relative lift, and the p-value from a two-proportion z-test along with a 95% confidence interval for the lift.
2. Test for sample ratio mismatch (SRM) both daily and overall. If you find any, suggest likely root causes and how to mitigate them.
3. Examine day-to-day heterogeneity; is pooling the data appropriate? Justify your answer using a fixed-effects versus random-effects framework.
4. List at least three pitfalls that apply to this window (for example, seasonality or a holiday effect on 2025-09-01, novelty effects, user overlap, peeking). Propose guardrails you would put in place before launch.
5. If your minimal detectable effect (MDE) was a +5% relative lift over baseline, estimate the achieved power and decide whether you would roll out, iterate, or extend the test. State any assumptions.

Overview: This question assesses skill in A/B test analysis and experiment design, including conversion-rate calculation, statistical inference (two-proportion tests and confidence intervals), sample-ratio checks, heterogeneity assessment (fixed versus random effects), and power/MDE reasoning.

This question is drawn from an Amazon Data Scientist interview experience.