Design an A/B test for a new shop-ads algorithm

A new ranking or promotion algorithm will alter which shop ads appear and in what sequence. The question posed is: “How do we determine whether this new algorithm is good?”

Design an online experiment and analysis plan that covers the following:

1. Randomization / experiment unit

• Would you randomize by user, by session, or by another unit such as geography or device?
• What tradeoffs arise from interference, contamination, variance, returning users, and cross-session effects?

2. Primary and guardrail metrics

• Propose several plausible success metrics—for example CTR, CVR, revenue, advertiser ROI, long-term retention—and explain their tradeoffs.
• Include at least one user-experience guardrail and at least one marketplace-health guardrail.

3. Power / sample size

• Describe how you would conduct a power analysis, including which baseline rates are needed, what MDE means, and how to handle skewed revenue.
• Mention how you would handle heavy-tailed revenue, such as winsorization, log transformation, or CUPED.

4. Conflicting metric scenario

• Suppose CTR rises while revenue falls. List at least four plausible causes—both product and statistical—and explain how you would debug and decide.

5. Common experiment pitfalls

• List common pitfalls such as SRM, bots or fraud, novelty effects, duration, and multiple testing, and describe how you would monitor and mitigate them.

Overview: This question assesses experimental-design and data-analysis skills—specifically randomization and unit selection, metric definition and guardrails, power and sample-size reasoning, diagnosis of conflicting signals, and common experimentation pitfalls—within the Analytics & Experimentation category for a Data Scientist role.