ByteDance · Statistics & Data Analysis
Design an interference-robust A/B test for monetization
TrueInterview
October 7, 2026 · 1 min read
You are introducing a new tipping interface on creator (PGC/OGC) posts to drive monetization without sacrificing growth or traffic. Build an A/B test that can handle interference and supply–demand effects. Specify: 1) the randomization unit and clustering approach (for example, creator-level, ego-network, or geo-level) to reduce spillovers across users and posts; 2) eligibility and exposure rules that avoid treatment contamination between US and Asia time zones; 3) the primary metric hierarchy (such as payer conversion per DAU, ARPPU, creator revenue share) and guardrail metrics (retention, session length, abuse reports, ad revenue cannibalization); 4) power and duration targeting at least one weekly cycle, a ramp plan, and sample ratio mismatch detection with pre-registered stopping rules; 5) variance reduction (CUPED covariates like pre-experiment spend and creator popularity) and cluster-robust inference; 6) decision thresholds, and how you would roll out if the treatment improves monetization but slightly hurts growth. Be precise about the exact formulas and the minimal detectable effect you are targeting.
Overview: This question tests experimental design and causal inference skills in Analytics & Experimentation, focusing on interference mitigation, clustering and randomization choices, eligibility and exposure rules, metric hierarchy and guardrails, power and duration planning, variance-reduction methods, and cluster-robust inference in a two-sided marketplace. It is often used to assess a data scientist's practical ability to balance monetization and growth trade-offs through careful statistical planning, pre-registered stopping rules, SRM detection, MDE and rollout decision thresholds, rather than only conceptual understanding.