Design and analyze ads A/B test this week

You run an online ads platform and are comparing a new ad scheduling policy (B) against the current policy (A). The date is 2025-09-01. The scheduled readout window is the most recent 7 days through today: 2025-08-26 through 2025-09-01, inclusive.

Constraints and context:

• Users may appear on several platforms (web, iOS, Android) and in multiple time slots; some users receive more than one impression in a day.
• Primary KPI candidate: watch_time_per_impression (seconds). Guardrails: CTR, skip_rate, daily_active_users, and complaint_rate.
• There is known seasonality by day-of-week and time-of-day; some creatives are long-form videos.
• Offline conversions (site visits) are reported with a 24–48 hour delay.

Write a precise test plan and analysis procedure that addresses the following, with justifications:

1. Randomization unit and exposure control: should you randomize at the user, device, or impression level? How will you limit exposures and avoid cross-contamination across platforms and time slots? Specify the hashing/bucketing key.
2. Stratification and variance reduction: define strata (e.g., platform × day-of-week × time-slot) and state whether you will apply CUPED with a pre-period (provide exact pre-period dates). Define the CUPED covariate and show the adjusted estimator formula.
3. Metric definitions: formalize the primary KPI and guardrails (numerators/denominators), and state whether the analysis will be intent-to-treat. Explain how you will handle zeros and outliers (e.g., winsorization rules).
4. Tail choice and test: state whether the primary KPI hypothesis calls for a one-tailed or two-tailed test and why. Select an appropriate test (e.g., Welch’s t, stratified difference-in-means, or permutation) and show the test statistic you will use.
5. Sample size and stopping: compute or outline the required sample size per variant for a +5% relative lift in mean watch_time_per_impression, assuming a baseline mean of 42s and SD of 55s, alpha=0.05 (two-sided), and power=0.8. Describe any sequential monitoring rule (e.g., always-valid methods) if interim looks are planned.
6. Readout: define the exact 95% CI you will report, how you will pool across strata, and how you will adjust for multiple guardrails (e.g., Holm). Include at least two diagnostic checks for randomization balance and two for seasonality/novelty effects.
7. Sensitivity: describe how you would re-run the analysis if offline conversions are incomplete for the last 48 hours, and how that affects the readout window.

Overview: This question assesses experimental design and applied statistical analysis skills for A/B testing, including randomization, exposure control, stratification, variance reduction, metric formalization, hypothesis testing, sample sizing, sequential monitoring, diagnostics, and handling delayed offline conversions.