Estimate Experiment Duration with Power Analysis

Design a two-sided A/B test around a binary conversion metric. The control converts at $10\%$, the smallest lift worth catching is a rise to $11\%$, the significance level is $5\%$, the target power is $80\%$, traffic splits evenly between the arms, and $20{,}000$ eligible users arrive each day.

Work out the sample size that is required and the nominal run length. Then describe which operational inputs can stretch the real duration past that estimate, and what has to be settled before any power calculation is carried out.

Constraints & Assumptions

• Every user is assigned a single time and supplies one binary outcome.
• A standard normal approximation applies to two independent proportions.
• The one percentage point shift is treated as an absolute minimum detectable effect.
• Finite-population correction is set aside.

Clarifying Questions to Ask

• Does the hypothesis run one-sided or two-sided?
• Must $80\%$ power hold exactly at the minimum effect, or across a range of scenarios?
• What share of eligible users actually enter the experiment and go on to produce a mature outcome?
• Is there clustering, repeated exposure, more than one variant, or any planned interim look?

hint Convert sample size to calendar time last First derive observations per arm from alpha, power, the baseline variance, and the minimum detectable effect; only after that divide by effective daily traffic.

What a Strong Answer Covers

• The inputs that feed a power analysis, plus the difference between absolute and relative effects.
• An approximate sample-size computation for two proportions.
• Per-arm versus total sample size, and how traffic is allocated.
• Trigger rate, attrition, outcomes that arrive late, design effect, and full-cycle limits.
• A duration recommendation that does not simply halt at the first significant result.

Follow-up Questions

• What happens to the duration if the trigger rate is $20\%$?
• What adjustment is required when randomization happens at the household level?
• Why might a test keep running through whole weekly cycles even after the sample size is reached?

Overview: Work out the sample size and runtime for an A/B test on a binary metric. Tie alpha, power, the baseline rate, and the minimum detectable effect to traffic volume, triggering, clustering, and calendar limits.