Design an experiment to evaluate an onboarding progress bar

Scenario

In an onboarding flow similar to QuickBooks, the product team introduces a progress bar. The onboarding consists of four steps:

1. Onboard
2. Send payment
3. Set up bank account
4. File the tax

Question

How would you assess whether the progress bar actually helps? Your response should cover:

• The hypothesis and what you mean by “helps”
• Primary metric, diagnostic metrics, and guardrails (with tradeoffs)
• Experiment design (randomization unit, high-level sample size/power considerations, duration)
• Key pitfalls (novelty effects, instrumentation, segment heterogeneity) and how you would address them Overview: This question tests a candidate’s skills in experiment design, A/B testing, choosing metrics and weighing trade-offs, causal inference, and product analytics for measuring the effect of a UI element such as a progress bar.