How to estimate a feature’s causal impact on time spent

Imagine you work on a game platform similar to Roblox. A new product change (a “feature”) has been shipped, and you want to quantify its causal effect on user engagement, captured as daily minutes spent per user-day.

The rollout, however, is not completely random:

• The feature was turned on earlier for certain platforms or regions and later for the rest.
• Users who are already more engaged may be more likely to get or activate the feature sooner.
• Time spent shows strong day-of-week seasonality and an overall upward or downward trend.

You have event-level logs that have been aggregated into a user-day table:

• user_id
• date (in UTC)
• minutes_spent
• feature_enabled (1 if the feature is enabled for that user on that date)
• User attributes (for example, country, platform, account_age_days)

Task:

1. Explain how you would assess whether the feature changes minutes_spent, including your approach to confounding.
2. If you use Difference-in-Differences (DiD), specify:
   • The treatment and control groups and the pre/post windows.
   • The key identifying assumption(s) and how you would check them.
   • The regression specification you would run and which coefficient answers the question.
3. Identify common failure modes (such as assumption violations or interference) and at least two alternative approaches if DiD is not credible.

Also state any additional assumptions you need and which outputs (tables or plots) you would present to stakeholders.

Overview: This question tests causal inference and time-series experimentation skills—particularly dealing with confounding, staggered rollouts, seasonality, and credible identification of treatment effects—within the Analytics & Experimentation area at a mid-to-senior data scientist level.