Design and analyze a banner A/B test

You need to decide whether a home-page banner should be introduced. Plan and analyze the A/B test from beginning to end:

1. Randomization unit and exposure (user-level versus session-level); cover cross-session consistency and interference.

2. Specify the primary and guardrail metrics, such as CTR, post-click dwell time, retention, and revenue per session. Give a precise definition of accidental clicks and explain how to remove or reweight them (for example, dwell under 500 ms or returning within 2 seconds).

3. Powering: starting from a baseline CTR of 1.5% and an expected relative lift of 10%, calculate the required per-arm sample size for 90% power and $\alpha = 0.05$ (two-sided); show the formulas and assumptions (pooled variance, continuity correction optional).

4. Analysis: account for multiple banner placements (multiple comparisons), position bias, and novelty effects; state whether you would use CUPED or pre-period covariate adjustment.

5. Diagnostic checks: exposure logging, ratio checks, bot filtering, and sequential monitoring with appropriate $\alpha$ spending.

6. Decision rule: give explicit launch criteria and a fallback plan if the accidental-click rate spikes.

Overview: The question assesses how well a candidate can design and analyze A/B experiments, spanning randomization and exposure choices, precise metric definitions and guardrails, sample size and power calculations, analysis plans for multiple comparisons and covariate adjustment, and diagnostic validation in the Analytics & Experimentation area. It appears often because it examines conceptual understanding as well as hands-on use of experimentation methods—measuring the ability to handle measurement problems, statistical assumptions, and clear decision rules that establish whether product changes are supported by evidence.