Pinterest · Statistics & Data Analysis

Design metrics and experiment for Shopping launch

TrueInterview

October 7, 2026 · 1 min read

You are rolling out a new Shopping module placed within the Pins feed. Design an experiment and metrics plan that:

Pick one primary success metric and 3–5 guardrail metrics. For each, define it precisely—numerator/denominator, unit of analysis, aggregation window—and include at least one intermediate or funnel metric, such as num_clicks_of_new_feature or stay_time_in_shopping. Weigh the tradeoffs of using DAU and user time spent as primaries against alternatives such as Shopping CTR, Add-to-Cart Rate, or GMV/DAU, and state the acceptable direction and magnitude of movement for each guardrail.
Address spillover or interference (repins and shares may expose control users) and learning or novelty effects. Propose and justify one concrete design option: user-level randomization with exposure logging and adjacency tests; cluster or geo randomization; or a switchback (time-based) or two-stage saturation design. Specify the randomization unit, eligibility and exposure rules, cooldown, novelty burn-in, and how you would detect and quantify spillover (for example, through graph distance or household/geo adjacency) and learning (for example, through the time-on-feature slope).
Specify power and MDE: baseline rate assumptions, variance, intra-cluster correlation if clustering is used, horizon length, and how you would handle seasonality and spiky traffic such as weekly cycles. Include an AA test and a CUPED or covariate adjustment plan.
Define a decision framework for cases in which other metrics drop. Suppose that after a 21-day test you observe a +2.3% lift in Shopping CTR, +1.1% in GMV/user, −0.6% in overall time spent, and −0.2% in DAU. Describe your net weighted lift or multi-objective rubric, guardrail thresholds, sensitivity to long-term effects, and what you would recommend to the PM. Also include which additional diagnostics you would run before rollout, such as heterogeneity across user or creator segments, cannibalization of ad revenue, and repeat usage versus one-off novelty.

Overview: This question assesses a data scientist’s competency in experimental design and product analytics, including metric selection and precise definitions, guardrail and funnel diagnostics, spillover and novelty handling, power/MDE estimation, and multi-objective decision framing.

Pinterest · Statistics & Data Analysis

Design metrics and experiment for Shopping launch

TrueInterview

October 7, 2026 · 1 min read

You are rolling out a new Shopping module placed within the Pins feed. Design an experiment and metrics plan that:

Pick one primary success metric and 3–5 guardrail metrics. For each, define it precisely—numerator/denominator, unit of analysis, aggregation window—and include at least one intermediate or funnel metric, such as num_clicks_of_new_feature or stay_time_in_shopping. Weigh the tradeoffs of using DAU and user time spent as primaries against alternatives such as Shopping CTR, Add-to-Cart Rate, or GMV/DAU, and state the acceptable direction and magnitude of movement for each guardrail.
Address spillover or interference (repins and shares may expose control users) and learning or novelty effects. Propose and justify one concrete design option: user-level randomization with exposure logging and adjacency tests; cluster or geo randomization; or a switchback (time-based) or two-stage saturation design. Specify the randomization unit, eligibility and exposure rules, cooldown, novelty burn-in, and how you would detect and quantify spillover (for example, through graph distance or household/geo adjacency) and learning (for example, through the time-on-feature slope).
Specify power and MDE: baseline rate assumptions, variance, intra-cluster correlation if clustering is used, horizon length, and how you would handle seasonality and spiky traffic such as weekly cycles. Include an AA test and a CUPED or covariate adjustment plan.
Define a decision framework for cases in which other metrics drop. Suppose that after a 21-day test you observe a +2.3% lift in Shopping CTR, +1.1% in GMV/user, −0.6% in overall time spent, and −0.2% in DAU. Describe your net weighted lift or multi-objective rubric, guardrail thresholds, sensitivity to long-term effects, and what you would recommend to the PM. Also include which additional diagnostics you would run before rollout, such as heterogeneity across user or creator segments, cannibalization of ad revenue, and repeat usage versus one-off novelty.