How should you evaluate unconnected content?

A social media platform has introduced a feed change that increases the share of unconnected content—posts from creators with no existing social relationship to the viewer. The stated goal is to improve engagement, but the interviewer wants a broader product and experimentation framework rather than a single engagement-rate metric.

Respond to the following as a data scientist:

1. How would you define success for this launch?

   • Separate short-term engagement, meaningful social interaction, long-term retention, and creator or ecosystem health.
   • State which metric you would choose as the primary decision metric and why.

2. What metric framework would you use?

   • Lay out a funnel from impression to consumption to interaction to downstream value.
   • Include primary metrics, secondary metrics, and guardrail metrics.
   • Discuss the tradeoffs between rate-based metrics and absolute metrics.

3. If unconnected content does not directly increase social interaction between friends, why might the company still want to launch it?

   • Provide product, business, and user-value hypotheses.

4. What are the main risks of this launch?

   • Consider cannibalization of friend content, low-quality recommendations, safety concerns, creator concentration, heterogeneous user effects, and misleading aggregate metrics.

5. How would you design an experiment to evaluate this launch?

   • Specify the unit of randomization, treatment definition, duration, power or MDE considerations, segmentation strategy, and how you would handle interference or network effects.
   • Explain what could go wrong in measurement and interpretation.

Overview: This question evaluates a data scientist's competency in product analytics, experimentation design, metric engineering, causal inference, and risk assessment for feed and recommendation changes.