Meta · Statistics & Data Analysis

Design metrics and geo A/B for new feature

TrueInterview

October 7, 2026 · 2 min read

You are proposing a new Marketplace feature called Verified Seller Badges, designed to boost buyer trust and monetization while avoiding harm to the user experience.

Mission and hypotheses: State your mission in precise terms, and write primary and secondary hypotheses that can be falsified.
Metrics: Define a North Star Metric (for example, weekly GMV per active buyer) along with 4–6 supporting metrics, at least two of which should be counter or guardrail metrics (such as fraud reports per 1,000 transactions, session crash rate, or ad revenue per session). For each metric, explain why it is diagnostic and how to compute it at both user-level and geo-level granularity.
Experiment design: Propose a geo-level clustered A/B test. Specify the cluster unit (city or metro), the stratification variables (e.g., active buyers, baseline GMV, seasonality, device mix), the matching strategy, the number of clusters per arm, the traffic ramp plan, the duration, and your approach to contamination, spillovers, and staggered rollouts.
Sample size and power: Show how you would estimate the minimum detectable effect for the NSM, including variance assumptions and any design effects introduced by clustering.
Instrumentation: List the exact events and attributes you need in logs to compute all metrics and diagnose the mechanism (for example, badge impressions, seller profile views, message initiations, purchase confirmations).
Decision framework: Suppose the test shows +2.0% (p<0.05) on the NSM, −0.3% (not significant) on sessions per user, and +0.8 bps in fraud reports (p=0.06). Explain the launch decision, including how you would incorporate engineering cost, staffing, and operational feasibility. Show a back-of-envelope estimate of potential incremental revenue assuming $0.50 revenue per incremental purchase and the observed lift.
External data: Name one third-party signal you might use to improve targeting, and discuss privacy and compliance considerations as well as how you would validate its incremental value without bias.

Overview: This question assesses experimental design, metric definition, and diagnostic analysis skills for marketplace features, including geo-clustered A/B testing, hypothesis formulation, power/MDE calculations, instrumentation, contamination handling, privacy considerations, and incremental revenue estimation.

Meta · Statistics & Data Analysis

Design metrics and geo A/B for new feature

TrueInterview

October 7, 2026 · 2 min read

You are proposing a new Marketplace feature called Verified Seller Badges, designed to boost buyer trust and monetization while avoiding harm to the user experience.

Mission and hypotheses: State your mission in precise terms, and write primary and secondary hypotheses that can be falsified.
Metrics: Define a North Star Metric (for example, weekly GMV per active buyer) along with 4–6 supporting metrics, at least two of which should be counter or guardrail metrics (such as fraud reports per 1,000 transactions, session crash rate, or ad revenue per session). For each metric, explain why it is diagnostic and how to compute it at both user-level and geo-level granularity.
Experiment design: Propose a geo-level clustered A/B test. Specify the cluster unit (city or metro), the stratification variables (e.g., active buyers, baseline GMV, seasonality, device mix), the matching strategy, the number of clusters per arm, the traffic ramp plan, the duration, and your approach to contamination, spillovers, and staggered rollouts.
Sample size and power: Show how you would estimate the minimum detectable effect for the NSM, including variance assumptions and any design effects introduced by clustering.
Instrumentation: List the exact events and attributes you need in logs to compute all metrics and diagnose the mechanism (for example, badge impressions, seller profile views, message initiations, purchase confirmations).
Decision framework: Suppose the test shows +2.0% (p<0.05) on the NSM, −0.3% (not significant) on sessions per user, and +0.8 bps in fraud reports (p=0.06). Explain the launch decision, including how you would incorporate engineering cost, staffing, and operational feasibility. Show a back-of-envelope estimate of potential incremental revenue assuming $0.50 revenue per incremental purchase and the observed lift.
External data: Name one third-party signal you might use to improve targeting, and discuss privacy and compliance considerations as well as how you would validate its incremental value without bias.