Define success metrics for Instant Book

Thumbtack is evaluating an “Instant Book” feature that would let customers reserve a pro immediately at a pre-agreed price and time, rather than waiting for quotes.

Answer:

1. North-star and objectives: Propose one north-star metric for marketplace health and two secondary objectives that reflect both the demand (customer) and supply (pro) sides. Define the exact formulas and units.
2. Diagnostic metrics: List at least six diagnostics spanning liquidity (time-to-first-commit), fulfillment rate, cancellations/reschedules, refund rate, average booking value, pro utilization, and fairness (for example, the Gini coefficient of bookings across pros within a category-region-week). Include target directions and acceptable short-term regressions.
3. Measurement plan: Describe the instrumentation/events, deduplication rules, and attribution windows. Make clear how you would separate organic bookings from Instant Book bookings, handle sessions that move across devices, and prevent double-counting when a request also receives quotes.
4. Rollout strategy: Recommend a staged launch (for instance, supply-gated by category-region with minimum active pros and SLOs), keeping a holdback to capture long-term effects. Explain quota controls so non-Instant-Book requests are not starved.
5. Success criteria and decision tree: Specify quantitative launch gates after 2 weeks and 6 weeks, including guardrails that must not worsen by more than X%.
6. Risks and mitigations: Identify at least five risks (such as adverse selection, schedule conflicts, price anchoring, unfair exposure, and fraud) and propose concrete mitigations and monitoring.

Overview: This question assesses a data scientist’s ability to define north-star and diagnostic metrics, design instrumentation and attribution, plan staged rollouts, and identify operational risks for a two-sided marketplace feature (metrics definition, experimentation design, measurement, governance, and risk mitigation).