Calculate a Confidence Interval

Imagine an online experiment that compares a treatment group and a control group on conversion rate. The treatment records x1 conversions among n1 users, while the control records x0 conversions among n0 users.

Demonstrate how to calculate a 95% confidence interval for the treatment effect manually.

• Give the point estimate for the absolute lift in conversion rate.
• Work out the standard error using an independent-samples approximation.
• Provide the formula for the 95% confidence interval.
• Describe how the result shifts when the interviewer wants relative lift rather than absolute lift.
• Say when the normal approximation is suitable and when an exact, score-based, or bootstrap interval would be preferable.

Overview: The item gauges a candidate's grasp of statistical inference for A/B tests, in particular how confidence intervals for differences in proportions are built and read, and it sits in the Statistics & Math area for Data Scientist positions; it probes hands-on work (obtaining point estimates and standard errors) together with conceptual grounding (the assumptions behind the normal approximation versus exact, score-based, or bootstrap intervals). Interviewers often pose it to see whether someone can quantify a treatment effect, judge when an approximation holds and what other interval methods exist, and tell absolute lift apart from relative lift when reporting results.