Interpret p-values and common pitfalls

A Fraud Data Science interview includes a set of p-value questions.

Respond to each item below within a fraud or experimentation setting:

1. Give a precise definition of a p-value. What does it tell you, and what does it not tell you?
2. Suppose you run an A/B test on a new friction step, such as an extra OTP, and obtain $p=0.03$. What can you conclude? What else do you need to know, including effect size, power, and business impact?
3. List at least four common mistakes people make with p-values in real analyses, such as multiple testing, p-hacking, peeking, nonstationarity, and selection bias.
4. Explain how you would change your analysis in each of these situations:
   • You are testing many regions or segments.
   • Outcomes are rare and arrive late, for example chargebacks that show up weeks afterward.
   • Randomization is imperfect, or there is interference/spillover between units.

Give at least one specific method for each case, such as Bonferroni/FDR, sequential testing, CUPED, Bayesian methods, or cluster-robust standard errors.

Overview: This question assesses how candidates interpret p-values, hypothesis testing, experimental design, and statistical pitfalls in a fraud or experimentation setting, including effect size, statistical power, multiple comparisons, delayed or rare outcomes, and imperfect randomization; category/domain: Statistics & Math.