Diagnose a sudden KPI drop and validate causes

One of your primary metrics, comments per DAU, takes a sharp, material fall. Lay out a disciplined plan for root-cause analysis and validation.

a) Scoping and sanity: Measure the decline in both absolute and relative terms, verify it consistently in dashboards and raw logs, and eliminate instrumentation problems (release notes, schema changes, missing events, clock skew). Which guardrail metrics do you check first, and why?

b) Slice-and-localize: Specify the precise segmentation you would apply (platform/app version, geography, cohort, time of day, traffic source, user tenure, content category, creators versus consumers) and the funnel stages you would examine. How do you distinguish a mix shift from degradation within a segment?

c) Attribution: List possible internal and external causes (code deployments or feature flags, A/B tests, recommender updates, moderation policies, outages; holidays or news events, competitor moves). For each one, suggest a validation approach: fast rollback, turning off a flag, switchback tests on suspect features, holdout comparisons, difference-in-differences against an unaffected market, or placebo tests.

d) Time-series methods: Choose an anomaly detection or causal impact method (for example, STL plus CUSUM, or BSTS/SCUL) with suitable controls. Which pre-treatment period and control series would you select, and how do you prevent post-treatment contamination?

e) Decision and follow-up: Set the thresholds for calling something a root cause rather than a correlation, describe the rollback or mitigation plan, and explain how you would track recovery and stop the issue from recurring (alerts, canary releases, automatic rollbacks, dashboards).

Overview: This question tests a data scientist's ability to perform root-cause analysis, verify metric instrumentation, localize problems by segment, attribute internal versus external causes, and apply time-series causal-impact and anomaly-detection techniques.