Investigate marketplace metrics and experiment rollout

The scenario is a Senior Data Scientist interview at a two-sided marketplace such as Instacart, where customers place delivery orders and shoppers decide whether to accept and complete them. Respond to the related interview questions below:

1. A core marketplace metric has fallen over the past two weeks. Describe a structured process for judging whether the fall is genuine and for narrowing down the likely causes. Your approach should take into account data-quality problems, seasonality, pricing or product changes, supply-demand balance, user/shopper/merchant segments, and outside influences such as competitors or weather.

2. The company is exploring a new pricing model that increases shopper pay during rush hours to motivate them to accept more orders. Because shopper behavior and customer demand interact inside each local market, an ordinary user-level A/B test can suffer from interference and network effects. Design an experiment. State the unit of randomization, how you would form matched or lookalike markets, the pre-period covariates and balance checks you would rely on, the primary success metrics, the guardrail metrics, and how you would analyze the results while addressing spillovers and seasonality.

3. Now suppose the experiment has ended. The pre-specified north-star metric is profit per order. The treatment raises average order volume, but profit per order declines. Should the company launch the pricing model? Explain how you would weigh a negative north-star metric against positive secondary metrics, which follow-up analyses are appropriate, and when deciding not to launch is still the correct conclusion.

4. A dashboard indicates that D14 retention dropped sharply for the latest week, while older cohorts appear stable. Explain how you would determine whether this is a genuine product issue or a false alarm caused by incomplete cohorts, delayed event ingestion, or metric-definition problems. Be explicit about cohorting, right-censoring, and the correct monitoring view to use.

Overview: This question tests capabilities in marketplace analytics, causal inference, experiment design, metric instrumentation, cohort and retention analysis, and reasoning about trade-offs between core business metrics and secondary signals.

This item is drawn from a community-shared Instacart Data Scientist interview experience.