Defining Marketplace Success Metrics and Diagnosing Their Movement

You are asked to define success metrics for a two-sided live-commerce marketplace. When a primary metric moves unexpectedly, the team needs a diagnosis that separates product effects from shifts in buyers, sellers, inventory, and measurement.

Boundaries and Assumptions

• A buyer may purchase from several sellers, and a seller may offer several items.
• The metric could be a ratio, so the numerator and denominator need to be examined separately.
• Promotions and the mix of supply can shift at the same time as product behavior changes.
• Late transactions and refunds can revise values that were already recorded historically.

Clarifying Questions to Raise

• Which marketplace decision is this metric meant to inform?
• Does success mean near-term conversion, healthy repeat usage, liquidity, or contribution margin?
• At what level of granularity and after what delay are orders, refunds, sessions, and listings treated as final?

Part 1 — Metric Design

Propose a single primary marketplace metric plus a small set of guardrails covering buyers, sellers, liquidity, quality, and economics.

What This Part Should Include

• A clear unit, denominator, time window, and rule for what is included
• Metrics linked through a causal funnel
• Counter-metrics that guard against optimizing only one side

Part 2 — Diagnosing the Change

Describe the query and analysis sequence you would follow when the primary metric changes suddenly.

What This Part Should Include

• Instrumentation and definition checks before jumping to causal explanations
• Decomposition of the numerator and denominator
• Cohort, mix, supply, promotion, and latency analysis

What a Strong Response Includes

• Exact metric definitions
• A diagnostic tree that can be falsified
• How refunds and late-arriving data are handled

Hint — Break down before slicing: For a ratio, first check whether the numerator moved, the denominator moved, or both. Then keep definitions fixed and decompose by stable cohorts and marketplace sides.

Follow-Up Questions

• How would you detect that a metric improved because low-converting users were starved?
• Which metric should be used when computing experiment power?

Overview: This is a product analytics interview about defining success for a two-sided marketplace and diagnosing an unexpected metric change. It assesses metric decomposition, cohort and funnel analysis, instrumentation checks, and the ability to separate product effects from mix shifts.