Design an A/B for ATO rule

Experiment design case: PayPal/Venmo is preparing to roll out a new real-time ATO rule that blocks high-risk transfers. Design and analyze an online experiment to estimate the net business impact.

Constraints and inputs: (1) The randomization unit must avoid cross-over: decide between user_id and transaction-level, and justify your choice using interference risks (recipients may be shared across arms). (2) Baselines: weekly fraud base rate on transfers $p_0 = 0.0012$, average fraudulent loss per incident $L_f = \$200$, expected relative reduction = 20%; legitimate block cost $C_{fp} = \$1.50$ per blocked legit transfer; expected block rate under treatment = 1.0% of legit transfers. (3) Traffic: 10M transfers/week, average 5 transfers per active user/week; ICC (cluster at user) = 0.02, average cluster size $m = 5$. (4) Guardrails: authentication success rate, dispute rate within 7 days, P95 time-to-pay. (5) Statistics: two-sided $\alpha = 0.05$, power 0.80; allow sequential monitoring (daily) with alpha spending; require pre-registration and an A/A test.

Tasks: A) Choose the randomization unit and explain spillover/contamination mitigations (e.g., recipient or graph clusters). B) Compute the minimum per-arm sample size (transfers) for detecting a 20% relative drop in fraud rate using a two-sample proportion Z-test; then inflate by the design effect $DE = 1 + (m-1) \times ICC$. Show formulas and numeric results. C) Convert the detectable effect into expected weekly net dollars using: $\text{Net} = (\text{Fraud prevented} \times L_f) - (\text{Incremental legit blocks} \times C_{fp})$; state any additional assumptions you need and bound the estimate. D) Define primary, secondary, and guardrail metrics with precise denominators; specify slicing (e.g., by device novelty, account age). E) Propose a ramp plan and stopping rules under a group sequential design (e.g., Pocock or O'Brien–Fleming), and how you will monitor production for post-launch drift.

Overview: This question assesses a data scientist's experiment-design and statistical-analysis skills, including cluster-aware randomization, power/sample-size calculation, sequential monitoring, and translating detectable effects into net business impact for a real-time account-takeover prevention rule on a payment platform.