Design Testing Without A/B Experiments

Imagine a product team needs to assess a new feature meant to boost engagement and long-term retention, but a clean randomized A/B test isn't possible due to legal, engineering, or rollout limits. How would you judge whether the feature actually benefits the business? Cover all of the following:

1. Specify the business objective, unit of analysis, treatment, and primary success metric.
2. Suggest primary and guardrail metrics—for instance, click-through rate, session depth, 7-day retention, latency, complaint rate, and revenue per active user—and discuss the trade-offs among them.
3. If randomization is not an option, compare several counterfactual or causal inference approaches, including difference-in-differences, synthetic control, matching, propensity scores, inverse probability weighting, regression adjustment, doubly robust estimation, instrumental variables, and regression discontinuity.
4. For each approach, describe the key assumptions, likely sources of bias, and how you would check those assumptions in practice.
5. Explain how your approach would shift for an opt-in feature, a one-sided rollout, a staggered launch across regions, or a policy change that affects all users at once.
6. If partial randomization is available, explain whether you would choose a switchback, geo experiment, or phased rollout, and how power, MDE, and variance reduction techniques such as CUPED would factor in.
7. Finally, suppose the core KPI drops sharply on a single day after launch. Walk through how you would determine whether the drop stems from the feature, instrumentation problems, traffic-mix shifts, outages, seasonality, or another external cause.

Overview: This question tests a data scientist's command of causal inference, observational study methods, metric definition, experimental design trade-offs, and failure-mode analysis when randomized A/B testing is not feasible.