How would you test swapping two CTA buttons?

You are responsible for the Airwallex marketing/product landing page. The page has two main call-to-action buttons:

• “See a demo”
• “Get started”

A proposal recommends swapping their order (for example, the left/right placement or primary/secondary styling remains unchanged, but the sequence changes).

Questions:

1. What user or business problem might this change be intended to solve? Give 2–3 plausible explanations.

2. Create an experiment (or evaluation plan) to decide whether to launch it:

   • What is the primary success metric, and why?
   • Which secondary/diagnostic metrics would you monitor to understand tradeoffs throughout the funnel?
   • Which guardrail metrics are important, particularly for a fintech onboarding flow?

3. What are the main pitfalls and biases to look out for (for example, novelty effects, cannibalization between CTAs, heterogeneous effects by segment)?

Assume you can run an A/B test on page traffic. Specify the unit of randomization and the analysis approach.

Overview:

This question tests A/B test design, causal inference, funnel metric selection, and diagnostic/guardrail reasoning in Analytics & Experimentation for a Data Scientist role. It is often asked to assess how candidates think about user behavior, conversion trade-offs, and risk management in sensitive areas such as fintech.