How do you diagnose a ratio metric change

Suppose an A/B test finds a statistically significant lift in a ratio metric for the treatment group:

• $CTR = \frac{clicks}{impressions}$ rose by +1.2% relative to control.

Product stakeholders are nevertheless unsure whether this reflects a genuine improvement or just an artifact.

Describe how you would investigate why CTR moved and decide whether the change can be trusted. Your response should cover:

1. Breaking the ratio into movements in the numerator and denominator.
2. Composition effects (such as shifts in traffic mix) and Simpson’s paradox.
3. Guardrail or invariant checks for instrumentation problems or ranking changes.
4. Statistical issues specific to ratio metrics (for example, variance estimation, delta method versus bootstrap).
5. Which follow-up analyses or further experiment iterations you would run.

Overview: Assesses diagnostic and statistical thinking for experiment analysis—in particular, decomposing ratio metrics into numerator and denominator movements, spotting composition effects and Simpson’s paradox, checking instrumentation and guardrails, and choosing suitable variance estimation methods.