Evaluate Facebook Dating launch and validate success

Suppose you must decide whether to take Facebook Dating from a small market pilot to a wider launch. Put together a rigorous validation plan:

• Hypotheses and success metrics: Set the primary objective (for example, successful matches per active seeker), the guardrails (retention in the core app, abuse reports on messages, privacy complaints), and the counter-metrics (whether core social engagement gets cannibalized). Spell out how each metric is defined and what deltas you will tolerate.
• Experimental design: Lay out a market-level rollout test (a geo ramp) against user-level randomization. Justify the unit of randomization, the spillover risk, and your mitigation for cross-market contamination (for instance, geo fencing and intent-to-treat analysis).
• Power and duration: Sketch a rough sample size and timeline using plausible baseline rates and minimum detectable effects; explain how you will watch for novelty effects and winner's curse.
• Expected user trends: Describe the shapes you expect for new-user activation, 1/7/28-day retention, match-to-message conversion, and seasonality; which anomalies would concern you (for example, a sign-up spike with no matching, or drop-offs driven by gender imbalance)?
• Validation without a full RCT: If a randomized trial cannot be run, propose a quasi-experimental method (such as synthetic controls or staggered DiD with pre-trend checks). Detail the diagnostics you need before you trust the estimate.
• Scale/readiness criteria: Define the precise quantitative and qualitative gates for expanding, pausing, or rolling back; cover privacy/SOC2 readiness, abuse tooling, and on-call load.

Overview: This question tests a data scientist's grasp of product analytics, experimental design, causal inference, and launch validation, since it demands primary and guardrail metric definitions, decisions about test unit and spillover, power and duration planning, expected behavioral signals, quasi-experimental fallbacks, and scale/readiness gates. It comes up often in Analytics & Experimentation interviews because companies want confidence that a pilot can be scaled safely and dependably; it probes both hands-on use of experimental methods and conceptual command of statistical power, spillover risk, diagnostic checks, and operational issues like privacy and abuse monitoring.

Read the complete Meta Data Scientist interview write-up this question was drawn from

Community answers

Answer from jackiechan26

how can the primary metric be SM7/AS when the control group has no access to this feature?