Assess ranking change and design experiment

A product that supports multiple accounts per user currently sorts those accounts by when they were last visited. The team behind it wants to flip the ordering so that accounts carrying the largest number of notifications come first, on the theory that putting the account with the most waiting activity at the top makes people more likely to engage. When two accounts have the same notification count, the more recently visited one goes ahead of the other.

Suppose you have historical logs recorded at the user-session-account grain, capturing for every session:

• the accounts that were displayed, in the order they appeared
• how many notifications each account had when the session began
• when each account was last visited
• whether the user picked that account
• whether the user tapped into any notification
• later outcomes, such as conversion or purchase and retention at 7 days

How would you:

1. Draw on the historical data to judge whether this change to the ordering is likely to pay off, before it ships?
2. Lay out an A/B test that captures the causal effect of the new ordering?

In your answer, spell out:

• which users qualify for the analysis
• what gets randomized
• the primary metric or metrics, the secondary ones, and the guardrails
• how confounding and selection bias would be dealt with in the historical work
• what to do about sessions in which the old and new orderings come out the same
• how to reason about power, the minimum detectable effect (MDE), and how long the experiment runs
• the segments and edge cases worth examining

Overview: This item tests how well a candidate handles causal inference, experiment design, metric definition, and observational analysis for ranking and engagement problems, in the Analytics & Experimentation area for Data Scientist roles.