Compute p-values for 2 variants vs control

A/B/n test: calculating p-values and deciding whether to ship

You conducted an online A/B/n experiment with one control and two treatment variants (A/B/C). You receive an aggregated results table with one row per group: Table: ab_results

• group (STRING): either control, variant_1, or variant_2
• users (INT): count of unique users assigned to that group
• conversions (INT): count of users who converted, where the outcome is binary Assume the following:
• Assignment is independent, and every user belongs to exactly one group.
• The metric is conversion rate = conversions / users.
• The goal is to test whether each variant changes conversion rate relative to control.
• Use a two-sided test unless you can justify a one-sided test.

Tasks

1. In Python, calculate the p-value for:
   • variant_1 vs control
   • variant_2 vs control (State which statistical test you selected and why.)
2. Give 95% confidence intervals for the lift, defined as the difference in conversion rates, for each variant compared with control.
3. Since there are two comparisons against the same control, explain how you would address multiple testing (for example, Bonferroni, Holm, FDR) and how that changes your decision.
4. Interpret the results in plain language and make a ship / no-ship recommendation, including the conditions under which you would run a follow-up experiment. Include any assumptions or caveats, such as sample size adequacy, novelty effects, missing data, or metric definition issues. Overview: This question assesses a candidate's ability in statistical hypothesis testing and experiment analysis, specifically A/B/n comparison of conversion rates, p-value calculation, confidence interval estimation for lift, multiple-testing considerations, and interpretation of results.