Experiment on increasing order notifications

Context: A notifications team is considering sending more order-related push messages to boost sessions and conversions.

Tasks:

1. Design an experiment that measures the incremental effect of raising notification frequency, improving send timing, or both. State the randomization unit (user), exposure caps, time-of-day stratification, and controls for cross-channel effects from email and SMS. Discuss multi-arm or bandit variants, and when they should or should not be used.
2. Specify primary success metrics — incremental orders, revenue, profit — and guardrails such as opt-out rate, uninstall rate, complaint rate, and session quality. Cover both per-user and per-notification metrics and how they will be aggregated.
3. Measure long-term effects: create a holdout or staggered rollout with a 4–8 week follow-up, accounting for novelty effects, fatigue, and decay. Explain how you would estimate persistent lift, for example through switchback tests, long-lived holdouts, or synthetic controls.
4. Handle interference and repeated exposure: propose an analysis that accounts for saturation and diminishing returns, such as dose-response curves, exposure-weighted treatment, or instrumental variables based on randomized send/no-send decisions at the trigger level.
5. Define power and MDE assumptions, stopping rules, and a decision framework that balances short-term lift against long-term retention risk. Include a concrete rollback criterion.

Overview: This question evaluates a data scientist's skills in experimental design, causal inference, metric definition and aggregation, power and MDE calculation, and cross-channel attribution for order-related messaging campaigns.