Decide when CTR falls but revenue rises

You are running an A/B test on ads ranking. The user-level results for treatment versus control are: CTR −3.0% (p=0.02), CPM +6.0% (p=0.04), impressions/user +1.5% (p=0.08), revenue per thousand impressions (RPM) +4.0% (p=0.05), and purchase conversion on click −0.8% (p=0.20). (1) Build a decision framework for a ship/hold recommendation using a north star metric (such as revenue or advertiser value) and guardrails covering user experience and integrity. (2) Compute the net revenue change per 1M impressions and break the drivers into a waterfall across CPM, CTR, and CVR. (3) List additional diagnostics—pacing, bid landscape shifts, supply mix, user segments—to detect Simpson’s paradox. (4) When presenting to the CFO, which visuals would you put on a single slide (for example, a forest plot of segment effects with confidence intervals, a driver waterfall, a traffic allocation/SRM chart), and what headline would you use?

Overview: This question tests knowledge of A/B testing, auction-based advertising metrics, how to decompose metrics to quantify revenue impact, causal diagnostics for hidden heterogeneity such as Simpson’s paradox, and executive-level presentation of results in the Analytics & Experimentation area.