Snapchat · Statistics & Data Analysis
Design an experiment for spam filtering impact
TrueInterview
October 7, 2026 · 1 min read
You are preparing to ship a stricter spam filter and need to measure how it affects friendship dynamics and same-day acceptance.
a) Lay out explicit hypotheses: a primary one about the effect on same-day acceptance rate, plus at least two guardrail metrics (such as total requests sent, approval latency, and false-positive spam rate). State the null and alternative precisely.
b) Propose an experimental unit and randomization design that reduces network interference (for example, clustering by requester or by recipient). Explain why you chose it and address possible spillover risks.
c) Define the primary and secondary metrics, including precise measurement windows and UTC date boundaries. Describe how to treat delayed approvals that cross day boundaries.
d) Outline your sample-size and power plan: the assumed baseline same-day acceptance rate, minimum detectable effect, source of variance, test duration, and a scheme for sequential looks that controls Type I error.
e) Describe how incomplete spam labels on users could bias your metrics, and give two mitigation strategies (for example, an unknown bucket plus sensitivity bounds, or propensity/inverse-probability weighting if the labeling is missing at random). Explain how you would present results after applying these adjustments.
Overview: This question tests experimental design and causal inference skills for estimating the effect of a stricter spam filter on same-day friend-request acceptance. It covers hypothesis specification, interference and randomization choices, metric definition and windows, power calculation, and handling incomplete labels.