Evaluate Fresh Content and Video Experiments

Pinterest aims to make the home feed feel fresher and more engaging. Respond to the interview questions below:

1. Propose a workable metric for content freshness. Clarify what should count as fresh, which user action or exposure to measure, and the time window you would use. Explain why that window is preferable to a shorter or longer one.
2. Describe the limitations of your freshness metric. What might it fail to capture? How could heavy users, light users, creators who post at different frequencies, and different content categories influence it?
3. Pinterest is thinking about showing more video pins in the home feed to raise engagement. State the null hypothesis and at least two alternative hypotheses, including one beneficial and one harmful alternative.
4. Assume an A/B test is randomized at the user level and runs for 14 days. Interpret the results below, including what the p-values mean and whether you would launch the change. Metric | Control | Treatment | Relative lift | p-value ---|---:|---:|---:|---: Users | 100000 | 100000 | - | - Home feed sessions per user | 5.00 | 5.03 | +0.6% | 0.08 Click-through rate | 4.00% | 3.92% | -2.0% | 0.03 Saves per 100 impressions | 1.20 | 1.26 | +5.0% | 0.01 Video watch time per user | 30.0 seconds | 33.0 seconds | +10.0% | <0.001 Hide or report rate | 0.50% | 0.53% | +6.0% | 0.04 7-day return rate | 42.0% | 41.8% | -0.5% | 0.20 Fresh impression share | 18.0% | 17.1% | -5.0% | 0.02
5. Explain how you would determine the sample size for an A/B test. How do baseline variance, minimum detectable effect, significance level, statistical power, traffic allocation, trigger rate, CUPED, and multiple metrics each affect power? Overview: This question assesses skills in metric design, experimental design, statistical inference, and product analytics: defining and critiquing freshness metrics, setting up null and alternative hypotheses for content changes, reading A/B test results, and computing sample size and power.