Diagnose and decide on watch-time drop

Hulu introduced a new homepage ranking model for all iOS traffic on 2025-08-25. From 2025-08-25 to 2025-08-31, average watch-time per session among new users on iOS declined by 2.7%, while Android remained flat. A major content premiere occurred on 2025-08-29, and a 20-minute payments outage on 2025-08-27 affected subscription starts. No holdout group was configured. Within 48 hours, recommend whether to roll back, continue, or apply mitigation.

Address the following:

• Specify the main decision metric(s) and guardrail metrics (such as session starts, error rate, churn proxy), and set explicit decision thresholds.
• Propose a way to identify the effect without a built-in holdout: (1) a difference-in-differences design that uses Android as the control and 2025-08-18–2025-08-24 as the pre-period; (2) a geo-based synthetic control within iOS. List the required assumptions (parallel trends, no interference), diagnostics, and falsification checks.
• Control for seasonality, the 2025-08-29 premiere, and the 2025-08-27 outage. Which fixed effects or control variables would you include? How would you address heterogeneity across countries, platform versions, and new versus returning users?
• Estimate the impact with confidence intervals and give a rough weekly revenue effect (making clear your assumptions for ad- and subscription-based revenue).
• Draft a quick remediation plan (such as revert, throttle, feature flags, guardrail monitors) and a forward plan for a sound experiment design in September (holdouts, pre-registration, CUPED, exposure logging).

Overview: This question tests a data scientist's abilities in causal inference, experiment analytics, defining metrics and guardrails, adjusting for confounders, and making rapid decisions under time pressure.