Design an experiment for pricing page redesign

A product group is reworking its pricing tier page. Until now the page listed only a monthly subscription; the new version introduces an annual option. You are the data scientist supporting this rollout.

Questions

1. Impact sizing: What are the key ways this change could affect the business, both positively and negatively, and how would you estimate the likely magnitude of each effect before running anything?

2. Metrics: Propose a metric framework:

   • Primary success metric(s)
   • Diagnostic metrics (to understand why it moved)
   • Guardrail metrics (to prevent harm)

3. Experiment design & ship decision: Explain how you would design the experiment and make a ship / no-ship decision, covering:

   • Unit of randomization and key segments to monitor
   • Sample size / power approach (MDE) and duration considerations
   • How you would analyze results (e.g., confidence intervals) and handle pitfalls (SRM, novelty, repeated exposure)

4. No experiment possible: If you cannot run an A/B test (policy, engineering constraints, or all-users launch), how would you measure impact as credibly as possible?

Overview: This prompt assesses a candidate's competency in experiment design, causal inference, impact sizing, metric framework development, and statistical analysis, all in the context of a pricing page redesign that introduces an annual subscription option.