Should you roll out if NSM decreases?

Scenario

You have completed an experiment. The north star metric (NSM) is profit per order.

Observed results

• Average order volume rose in the treatment group relative to control.
• Profit per order fell (either statistically significant, practically meaningful, or both).

Task

Would you launch this change? Walk through your decision process.

Requirements

Your response should address:

• Why the NSM deserves priority over secondary metrics.
• Which additional analyses you would perform (segment analysis, guardrail metrics, novelty effects, heterogeneous treatment effects).
• Any conditions under which you would still consider shipping (for example, if overall profit rises, long-term effects, strategic objectives).
• A definite final recommendation and next steps.

Overview: This question tests skill in experiment analysis, metric prioritization, and weighing trade-offs between a north star metric and secondary signals in A/B testing, covering core data science abilities in Analytics & Experimentation.