Choose KPIs and prove impact with experiments

Imagine you are on Expedia’s Search team, rolling out a new hotel-ranking model for enterprise clients whose priority is profitable bookings rather than clicks. a) Specify one primary KPI with an exact formula that matches client value—for instance, margin-adjusted bookings per search inside a 7-day attribution window—and provide at least three precise guardrail metrics with thresholds, such as cancellation rate $\le X\%$, p95 latency $\le Y$ ms, and diversity entropy $\ge Z$. b) Describe how you would check that offline proxy metrics, like NDCG@10 weighted by booking margin, actually track the online KPI; include the analysis plan, acceptable correlation or elasticity ranges, and what you would do if the two diverge. c) Lay out an A/B test design: pick the randomization unit (for example, search session), traffic allocation, ramp plan, MDE and sample-size derivation with stated base rates and variance assumptions, pre-experiment checks such as SRM, and a variance reduction technique like CUPED or stratification. d) Explain how you would identify and guard against metric gaming and novelty effects—clickbait and position churn, for instance—and define a rollback threshold. e) If bookings remain flat while cancellations increase 15% in the treatment group, state whether you would ship or roll back, supporting the call with expected profit impact and confidence intervals. f) Compute incremental gross profit per search with a 95% confidence interval, and explain how you would distinguish incrementality from attribution. g) For paid channels, recommend a multi-touch attribution method—Shapley/Markov, regression, or geo-experiments, for example—and describe how you would calibrate it against holdout or geo tests.

Overview: This item tests a data scientist's skills in KPI engineering, causal inference, experiment design, metric validation, and multi-touch attribution inside the Analytics & Experimentation area, with a focus on profit-driven search ranking.

See the full Expedia Data Scientist interview experience where this question originated.