Design and analyze an SBA mini case experiment

A team working on small-business acquisition is preparing a two-week pilot, under which new accounts are given a $100 credit after their identity is verified. The primary KPI is 30-day activation rate: the proportion of new accounts that spend at least $50 net on ads within 30 days after signup. Guardrail metrics are refund rate, support tickets per 1,000 accounts, average spend during the first 30 days, and CAC. Activation is $12\%$ at baseline. There is budget for 10,000 treatment accounts across 14 days, although daily traffic volume and account mix vary by region and industry. Build a rigorous experiment by choosing the randomization unit and stratification variables, handling day-of-week and regional seasonality, and planning for non-compliance and fraud. Determine the necessary sample size to achieve $80\%$ power at $\alpha=0.05$ while detecting a +2.0 percentage-point lift; state all assumptions and show the calculation method. Give the exact analysis plan: define ITT and TOT, describe variance reduction (for example, CUPED with pre-verification activity), set guardrail monitoring with kill-switch thresholds, and specify a precise decision rule such as a one-sided test with a superiority margin and multiplicity control. Describe how to inspect data quality (event lag, duplicate accounts, bots), measure treatment effect heterogeneity by industry, region, and spend propensity, and ensure the results generalize beyond the two-week window with novelty and seasonality adjustments. Finally, outline the rollout recommendation if observed lift is +1.6 percentage points, the 95% confidence interval is $[+0.2\text{pp}, +3.0\text{pp}]$, and support tickets go up by $12\%$.

The question assesses skill in experimental design and causal inference, including sample-size and power calculations, ITT/TOT analysis, variance reduction, heterogeneity measurement, data quality and fraud handling, guardrail monitoring, and product-experiment rollout decisions.