Compute p-values, power, and adjust errors

For each of the statistics questions below, provide exact calculations and interpret the results:

(a) After 5 interim looks, the data show Control with 12,000 users and 1,380 conversions, and Treatment with 12,200 users and 1,512 conversions. Calculate the two-sided p-value and a 95% confidence interval for the proportion difference. Then apply an O’Brien–Fleming alpha-spending adjustment for sequential monitoring, and explain whether the finding would remain significant without fully recomputing the spending functions if the exact schedule is unavailable.

(b) You monitor 5 metrics whose sorted p-values are $\{0.001, 0.012, 0.019, 0.070, 0.300\}$. Use Benjamini–Hochberg with FDR $q=0.05$ and identify which metrics are discoveries. Contrast this with Bonferroni at familywise $\alpha=0.05$.

(c) Bot detection: the prevalence is 5%. The model flags a user as a bot with FPR $=2\%$ and FNR $=10\%$. Given that a user is flagged, compute the posterior probability that the user is actually a bot. Show Bayes’ theorem with the numbers plugged in.

(d) Average Order Value (AOV) is right-skewed with mean $=32$, sd $=15$, and a heavy tail. Suggest a robust strategy for outlier handling and inference: compare the IQR rule, z-scores, and MAD/Huber M-estimators; recommend a log-transform with a delta adjustment and describe how to back-transform effects to the original scale using a smearing estimator.

Overview: This question tests statistical reasoning and applied inference, covering hypothesis tests for proportions under sequential monitoring, multiple-testing control (Benjamini–Hochberg versus Bonferroni), Bayesian posterior estimation for classification flags, and robust AOV estimation with transformations.