Intuit · Statistics & Data Analysis

Choose the right test for proportions

TrueInterview

October 7, 2026 · 1 min read

You are running an A/B test on the delivery completion rate (delivered_bool). The observed data are:

Control: 1,000 orders, 920 delivered, 80 not delivered.
Variant: 1,000 orders, 880 delivered, 120 not delivered.

Choose the most suitable significance test and explain why it is preferable to a t-test applied to proportions. Derive the test statistic formula by moving from the Bernoulli to the binomial to the normal approximation, and list the conditions required for that approximation to hold (expected cell counts, continuity corrections).
Calculate the p-value to three significant digits and a 95% confidence interval for the difference in proportions, showing the intermediate work.
Now suppose a small-market rollout produces: Control 20 delivered out of 25 total; Variant 12 delivered out of 18 total. Reassess which test to use (chi-square vs. Fisher’s exact vs. z-test). Compute the exact p-value or describe precisely how you would obtain it.
Explain when a two-sample t-test yields roughly the same result as a z-test for proportions, and when it breaks down. Include at least two concrete failure modes (for example, low counts, misuse of unpooled variance).

Overview: This question assesses understanding of hypothesis testing for binary outcomes, including choosing among the z-test, chi-square test, and Fisher’s exact test, deriving the test statistic from Bernoulli to binomial to normal approximations, and computing p-values and confidence intervals.

Intuit · Statistics & Data Analysis

Choose the right test for proportions

TrueInterview

October 7, 2026 · 1 min read

You are running an A/B test on the delivery completion rate (delivered_bool). The observed data are:

Control: 1,000 orders, 920 delivered, 80 not delivered.
Variant: 1,000 orders, 880 delivered, 120 not delivered.

Choose the most suitable significance test and explain why it is preferable to a t-test applied to proportions. Derive the test statistic formula by moving from the Bernoulli to the binomial to the normal approximation, and list the conditions required for that approximation to hold (expected cell counts, continuity corrections).
Calculate the p-value to three significant digits and a 95% confidence interval for the difference in proportions, showing the intermediate work.
Now suppose a small-market rollout produces: Control 20 delivered out of 25 total; Variant 12 delivered out of 18 total. Reassess which test to use (chi-square vs. Fisher’s exact vs. z-test). Compute the exact p-value or describe precisely how you would obtain it.
Explain when a two-sample t-test yields roughly the same result as a z-test for proportions, and when it breaks down. Include at least two concrete failure modes (for example, low counts, misuse of unpooled variance).