Design metrics and an A/B test for an app

Choose a consumer digital app you genuinely like. Treat the interviewer as if they have no prior familiarity with it.

1. Describe the product, its core jobs-to-be-done, the target audience segments, and its top three competitors, along with what sets the app apart.
2. List every existing revenue stream and the unit economics; then propose one additional revenue stream and build a six-month ROI model for it.
3. Define three exact, computation-ready success metrics for the new stream—each with its numerator, denominator, and event or window definitions—plus two guardrail metrics with thresholds.
4. List six specific product improvements for the app, then choose one to ship first and explain why.
5. Experiment design: set up an A/B test for the improvement you prioritized—specify the unit of randomization, eligibility and exclusions, assignment method, and how you would reduce bias. Using these inputs, calculate the minimum sample size and test duration: daily active users = $500{,}000$; baseline conversion-to-purchase per user-day = $15\%$; expected absolute lift = $+2.0$ percentage points; power = $80\%$; two-sided alpha = $5\%$; 1:1 split. State your assumptions and show the formula you would use.
6. If finance pushes back that the improvement is too expensive, present your cost-benefit model, the breakeven point, and the decision you would make if interim results fall short of the MDE after two weeks.

Overview: This question tests product analytics and experimentation skills—product framing, unit economics, revenue modeling, computation-ready metric design, statistical power and sample-size calculation, A/B test setup, and cost-benefit/breakeven analysis—for a Data Scientist role in the Analytics & Experimentation area, and it is often used to see whether candidates can turn product hypotheses into measurable business outcomes and defend their decisions under stakeholder pressure. It checks both conceptual understanding of segmentation, metric validity, and bias considerations and practical ability to define precise numerators and denominators, compute minimum sample sizes and test duration, and produce actionable ROI models.