Analyze an A/B test and present recommendation

Before your onsite interview, you receive an offline take-home project. Your job is to analyze an A/B test and present the results in a slide deck.

Suppose you are given a user-level table called experiment_events containing:

• user_id (string)
• variant (string; either 'control' or 'treatment')
• assignment_ts (timestamp, UTC)
• country (string)
• platform (string; one of 'ios', 'android', 'web')
• exposed (boolean; indicates whether the user actually saw the new experience)
• orders_7d (int; number of orders in the 7 days after assignment)
• revenue_7d (float; revenue in the 7 days after assignment)
• support_tickets_7d (int)
• is_new_user (boolean)

You also have pre-period covariates stored in user_pre_period:

• user_id
• orders_28d_pre (int)
• revenue_28d_pre (float)

Tasks:

1. Specify the primary metric and the guardrails you would rely on to decide whether to launch.
2. Describe the main statistical analysis you would run (tests/confidence intervals) and enumerate the sanity checks (such as sample ratio mismatch and covariate balance).
3. Explain how you would deal with noncompliance (some assigned users have exposed = false) and which estimand you would report: ITT or TOT.
4. Describe how you would examine heterogeneous treatment effects across platform, country, and new versus existing users without p-hacking.
5. Outline what your slide deck would include and how you would convey uncertainty and next steps to product managers and engineers.

Overview: This question tests experimental design, statistical inference, causal estimands, subgroup analysis, data quality checks, and communication with stakeholders in the analytics and experimentation space, aimed at an applied data scientist at the intermediate-to-senior level.