Present an A/B test project review

Ahead of the onsite, you finished a take-home assignment that analyzed an A/B test (assume standard product experimentation data: assignment, exposure, user events, and outcome metrics). In the onsite, you will present slides and respond to detailed follow-up questions.

What to prepare ahead of time

1. Recap the experiment's objective, design, and main assumptions.
2. Check experiment integrity and data quality (which checks would you perform?).
3. Measure the treatment effect on the metrics defined in advance.
4. Cover interpretation and limitations (confounding, interference, multiple testing, seasonality).
5. Give a definite ship/no-ship recommendation along with next steps.

Likely interviewer follow-ups

• How would you respond to a sample ratio mismatch?
• How do you choose between primary and guardrail metrics?
• How do you manage a large number of metrics or repeated peeks at the data?
• What if the overall impact is neutral but one segment improves substantially? Overview: This question tests a data scientist's skills in experimental design, data integrity validation, statistical estimation of treatment effects, and causal interpretation for A/B tests.