Design and evaluate a new group call feature

Product / Data Science Case: Group Calls for Messenger Groups

Messenger includes Groups, but group calls are not supported today. You are assessing whether to build and ship a group calling feature.

Respond to the following with both product sense and data science thinking:

1. Validating user need / problem

• How would you figure out if users truly need group calls, compared with alternatives such as group chat, voice notes, one-on-one calls, or outside apps?
• Which quantitative and qualitative signals would you rely on?

2. Selecting the supported group size

• How would you choose which group sizes to support (for example, a maximum of 4, 8, 16, or more participants)?
• What tradeoffs are involved, such as user value, engineering cost, quality and reliability, abuse or spam, and discoverability?

3. Measuring success Propose:

• a primary success metric, or a small set of them,
• diagnostic metrics,
• guardrail metrics. Explain your reasoning.

4. Experiment design with interference / network effects Since users are linked within groups, outcomes can spill over between treated and control users.

• If you randomize at the cluster level (for example, groups or user clusters), how would you:
  • define the clusters,
  • avoid clusters that remain connected and create spillover, or clusters that become too large,
  • analyze the experiment in a valid way?

5. A/B test plan and tradeoffs

• Propose an A/B test plan covering the randomization unit, duration, eligibility, and rollout.
• Discuss the main tradeoffs and possible failure modes, such as novelty effects, seasonality, infrastructure constraints, and measurement gaps.

Overview: This question tests a candidate's skills in product analytics, experiment design, causal inference with network interference, metric selection, and tradeoff analysis for launching group calls in a messaging product; it is aimed at the Analytics & Experimentation area for a Data Scientist position.