Roblox · Statistics & Data Analysis
Evaluate friend-interaction feature with network interference
TrueInterview
October 7, 2026 · 1 min read
You are preparing to launch a “friend interaction boost” that raises feed content ranking when friends engage with it. Since engagement spreads across the social graph, randomizing at the user level breaks SUTVA. Design an experiment that can credibly estimate the causal lift:
- Randomization unit: Pick and justify graph-cluster randomization (such as Louvain clusters) versus ego-network clustering versus geo/time switchbacks. How will you measure and limit the cross-cluster edge cut ratio and exposure contamination?
- Metrics: Define primary metrics (session time, meaningful interactions) and guardrail metrics (spam reports, creator revenue cannibalization). Specify exposure-weighted metrics for users who are only partially treated.
- Power: With average cluster size and intracluster correlation , derive the effective sample size . Show how this alters the required duration compared with user-level A/B testing.
- Analysis: Outline cluster-robust variance estimation, CUPED using pre-period outcomes, and intent-to-treat versus exposure-on-treated estimands. How do you handle creators whose audiences span treatment and control clusters?
- Diagnostics & fallbacks: Pre-specified spillover checks, negative controls, and a holdout of high-degree nodes. If contamination becomes too high mid-test, propose a redesign that preserves inference while limiting blast radius. Overview: This question tests a data scientist's ability in causal inference and network-aware experiment design, including randomization under interference, exposure-weighted metric specification, power estimation with intracluster correlation, and cluster-robust analysis.
Loading comments…