Formulate hypotheses and metrics for video-pin ramp

Pinterest aims to raise the share of video pins appearing in the home feed in order to improve engagement. Design the A/B test from start to finish.

Tasks:

• Specify exact null and alternative hypotheses for the primary metric; enumerate at least two alternative hypotheses reflecting possible trade-offs (such as higher engagement but lower quality), and indicate whether each is one-sided or two-sided.
• Identify the unit of experimentation, randomization scheme, exposure definition, and target population (for instance, all users versus new US users defined as signing up within 30 days of the action date). Discuss interference between users and constraints on content supply.
• Pick one primary metric and two to four secondary metrics, along with guardrail metrics (such as hide rate or complaint rate). Explain each metric's sensitivity and expected direction.
• List the segmentation dimensions (for example, pin_format, device, new versus existing users) and describe how you will address heterogeneous treatment effects while avoiding p-hacking.
• Outline the rollout plan, including holdout size, ramp schedule, and duration; the data quality checks for event logging completeness and attribution; and the stopping rules for early termination due to harm, futility, or success.

Overview: This question tests skills in experiment design and data science, including hypothesis formulation, metric selection and guardrails, segmentation, handling interference, rollout planning, power and minimum detectable effect considerations, and data quality monitoring, and it belongs to the Analytics & Experimentation area.