Meta · Statistics & Data Analysis

Run a clean A/B test for recommendations

TrueInterview

October 7, 2026 · 1 min read

You are required to run an A/B test for the new hashtag recommender beginning on 2025-09-01.

Choose the randomization unit—user, session, or impression—and defend that choice in light of possible interference, since users may view and follow the same hashtag in multiple sessions.
Describe how treatment and control groups are selected, the rollout or ramp plan, and the steps you would take to avoid crossover and contamination, such as sticky bucketing or holding out creators/hashtags where necessary.
Specify a primary metric, for example the 24-hour follow-through rate per exposed user, and list at least three guardrail metrics, such as session length, the violating-hashtag follow rate, and crash rate.
Power analysis: using a baseline follow rate of 4.0%, a two-sided $\alpha=0.05$ , power of 0.80, and a minimum detectable effect of +5% relative, calculate the required number of users per arm under user-level randomization and an intraclass correlation of $\rho=0.02$ due to repeated sessions. Show the formulas and adjustments for clustering, as well as for detecting expected sample-ratio mismatch.
Outline the bias controls you would use: CUPED or pre-period stratification, handling of novelty and winner’s curse effects, sequential monitoring rules, minimum test duration, and diagnostics for interference or network effects.

Overview: This question tests a data scientist’s ability in experimental design, causal inference, and statistical power analysis for A/B tests, including choice of randomization unit, prevention of contamination, selection of metrics and guardrails, clustering adjustments, and bias-control methods.

Meta · Statistics & Data Analysis

Run a clean A/B test for recommendations

TrueInterview

October 7, 2026 · 1 min read

You are required to run an A/B test for the new hashtag recommender beginning on 2025-09-01.

Choose the randomization unit—user, session, or impression—and defend that choice in light of possible interference, since users may view and follow the same hashtag in multiple sessions.
Describe how treatment and control groups are selected, the rollout or ramp plan, and the steps you would take to avoid crossover and contamination, such as sticky bucketing or holding out creators/hashtags where necessary.
Specify a primary metric, for example the 24-hour follow-through rate per exposed user, and list at least three guardrail metrics, such as session length, the violating-hashtag follow rate, and crash rate.
Power analysis: using a baseline follow rate of 4.0%, a two-sided $\alpha=0.05$ , power of 0.80, and a minimum detectable effect of +5% relative, calculate the required number of users per arm under user-level randomization and an intraclass correlation of $\rho=0.02$ due to repeated sessions. Show the formulas and adjustments for clustering, as well as for detecting expected sample-ratio mismatch.
Outline the bias controls you would use: CUPED or pre-period stratification, handling of novelty and winner’s curse effects, sequential monitoring rules, minimum test duration, and diagnostics for interference or network effects.