Estimate impact without experiments and pick variant

Part A — Estimating impact when an experiment is not possible

You are a Staff Data Scientist responsible for a product change (a feature, policy, or model update). Stakeholders want to measure the causal impact — the incremental lift — of that change, but you cannot run a randomized experiment (for example, legal restrictions, a requirement that every user receive the change, platform limitations, or risk).

Task:

1. Lay out an end-to-end approach for estimating the causal impact of the change from observational data (an ML-based counterfactual is fair game if it is appropriate).
2. State clearly:
   • The estimand (for example ATE, ATT, or incremental purchases per user/day).
   • The main assumptions needed for causal identification.
   • The major biases and failure modes (confounding, selection bias, interference, data drift, novelty effects, and so on).
   • How you would validate the approach (placebo tests, negative controls, sensitivity analysis, backtesting).
3. Explain how you would reason about short-term versus long-term impact, and what additional data or modeling you would need.

---

Part B — 3-variant experiment and forecasting post-launch conversion

You ran an online experiment with three variants (A/B/C). The goal is to maximize CTP (purchase rate), defined as: $\mathrm{CTP} = \frac{\#\text{purchases}}{\#\text{visits}}$ Observed results (assume visits are independent Bernoulli trials, with at most one purchase per visit):

• Variant A: 150 visits, 43 purchases
• Variant B: 200 visits, 48 purchases
• Variant C: 100 visits, 15 purchases

Questions:

1. Which variant is "winning"?
   • Give point estimates of CTP.
   • Quantify uncertainty (e.g., confidence or credible intervals).
   • Address multiple comparisons and decision criteria where relevant.
2. Assume you ship the chosen variant to 100% of traffic. How would you predict the future CTP after launch?
   • Describe a statistical approach for producing a forecast and an interval.
   • List the key factors that could make post-launch CTP differ from the experiment CTP (traffic mix shift, seasonality, ramp-up effects, novelty, instrumentation changes, and so on).
   • Mention how you would monitor and validate the forecast after launch (guardrails, alerting, recalibration).

Overview: This question assesses causal inference and experimental-analysis competence in Analytics & Experimentation and Data Science, spanning observational estimands, causal identification assumptions and biases, uncertainty quantification for A/B/C tests, multiple-comparisons reasoning, and post-launch forecasting and monitoring.