Design and Analyze Airbnb Locker Experiment

Airbnb is weighing a luggage locker feature that would let guests leave their bags before the official check-in time, rather than waiting for the room to be ready. You are the data scientist responsible for designing, and then analyzing, an experiment to determine whether Airbnb should ship this feature. The session has two parts: a case-study design conversation (Parts 1–5 and 7) and a hands-on analysis of a supplied experiment dataset (Part 6).

Constraints & Assumptions

• Airbnb operates as a two-sided marketplace: a change can affect guests, hosts, and platform revenue at the same time, and those interests may pull in different directions.
• Lockers are, at least in part, shared physical infrastructure with limited capacity in any given location.
• Outcomes cover the whole booking lifecycle: assignment can occur at booking, but important outcomes (post-stay rating, support contacts, damage claims) only become available after checkout.
• Unless you justify a different choice, assume conventional significance and power targets: $\alpha = 0.05$ (two-sided) and $80\%$ power.
• Historical baselines (such as support-contact rate and booking conversion) are available to size the test.

Clarifying Questions to Ask

• Is the locker presented before booking (a search/booking feature) or after booking (a check-in convenience)? That choice drives the primary metric and the randomization point.
• Are the lockers on-listing (host-operated, such as a smart lock box) or shared off-site infrastructure with pooled capacity? This determines whether SUTVA can hold at the reservation level.
• What is the planned launch mechanism — a UX toggle, a host opt-in, or a market-wide rollout? The experiment should match how the feature will actually be released.
• Is the feature free or paid? Pricing changes both the metric set and the revenue tradeoff.
• What baseline rates exist for the candidate primary metric, and what effect size would make the launch worthwhile?
• Are there safety, liability, or insurance constraints (lost or damaged luggage) that could block launch regardless of the metric result?

Part 1 — Randomization unit

Should randomization happen at the guest, reservation, host/listing, or market level? Justify your choice, and explain how the answer depends on exactly what the treatment changes.

Hint — What breaks independence?: The correct unit is the smallest level at which the Stable Unit Treatment Value Assumption (SUTVA) still holds — that is, one unit's treatment does not affect another unit's outcome. Ask whether a treated unit can contaminate a control unit. Two channels can do that here: shared physical capacity and a host changing behavior for everyone at a listing. Hint — Tie the unit to the mechanism: Match each candidate unit to the intervention it cleanly isolates: guest/reservation isolates demand and UX (does showing the option change behavior?); listing/host isolates a host-operated rollout; market/building isolates shared-capacity effects. If you cluster, you will later pay for it in sample size and inference.

What This Part Should Cover

• Argues from SUTVA, not from a default such as 'just randomize users'.
• Identifies both contamination channels — shared locker capacity and host-level behavior change — and which unit absorbs each.
• Links the unit choice to the launch mechanism and acknowledges the power vs. validity tradeoff (finer units are more powerful but riskier).

Part 2 — Eligibility, treatment, control

Define the eligible population, the treatment experience, and the control experience precisely enough for an engineer to implement the assignment logic.

Hint: Eligibility should be fixed before and independently of treatment (never on a post-assignment behavior such as 'guests who clicked the locker'). Consider who can even be offered a locker (geography, listing type, likelihood of arriving before check-in) and whether control is 'nothing' or 'today's host-managed luggage workaround.'

What This Part Should Cover

• Eligibility defined before assignment and independent of any post-treatment behavior.
• Treatment defined as the offer of locker access, not actual use.
• Control defined as the realistic status quo (the current host-managed workaround), not a strawman.

Part 3 — Metrics

Pick a primary metric, a set of secondary metrics, and guardrail metrics. Discuss the tradeoffs among guest experience, host experience, operational cost, safety, and marketplace revenue.

Hint — Pick primary by where the feature lives in the funnel: A pre-booking feature points toward a conversion/GBV primary; a post-booking convenience feature points toward a satisfaction or support-load primary. One sentence to anchor the choice: the primary should be sensitive to the feature, able to move within a realistic test window, and aligned with the launch goal. Hint — Guardrails protect the side you're not optimizing: Because this is a marketplace, include guardrails on the host/safety/cost side (cancellations, damage/lost-item claims, support cost per reservation, locker over-capacity) so a guest win that quietly harms hosts or trust-and-safety is caught.

What This Part Should Cover

• A single, well-justified primary tied to the feature's funnel position.
• A coherent metric tree (primary / secondary / guardrail), not a flat list of everything measurable.
• Explicit marketplace tradeoffs — names the guardrails that protect hosts, safety, and cost, and states a launch decision rule.

Part 4 — Sample size and duration

Explain how you would compute the required sample size and choose the experiment duration.

Hint — Sample size: For a binary primary (such as support-contact rate), use a two-proportion power calculation driven by a pre-registered minimum detectable effect (MDE). If you randomize at a cluster (listing/market) rather than the reservation, inflate $n$ by the design effect $\text{DEFF} = 1 + (\bar{m}-1)\rho$, where $\bar{m}$ is average cluster size and $\rho$ is the intraclass correlation. Hint — Duration is not just n / traffic: Duration must also cover weekly seasonality (weekday/weekend travel) and the booking-to-checkout lag, since post-stay outcomes only mature after guests leave. Low-traffic markets may constrain the timeline more than the math does.

What This Part Should Cover

• Sample size driven by a pre-registered MDE and the correct two-proportion power formula (for a binary primary).
• Design-effect inflation when randomizing at a cluster, with the right intuition for $\rho$ and cluster size.
• Duration reasoning that goes beyond $n/\text{traffic}$: weekly seasonality, the booking-to-checkout maturation lag, and thin-market constraints.

Part 5 — Analysis plan

Describe how you would estimate the treatment effect and compute a p-value for the primary metric, including how you would reduce variance and handle clustered randomization.

Hint — Estimand first, test second: Default to intent-to-treat (compare by assignment, not by who actually used the locker) so self-selection does not bias you. Then choose a test matched to the metric type (two-proportion z / logistic for binary; t-test / regression / bootstrap for continuous), and report the CI and effect size, not just $p$. Variance reduction: covariate adjustment / CUPED on pre-period behavior; if clustered, cluster-robust SEs at the randomization level.

What This Part Should Cover

• States the ITT estimand explicitly and never conditions on post-treatment usage.
• Matches the test to the metric type and reports a CI / effect size, not just a p-value.
• Variance reduction (covariate adjustment / CUPED) and cluster-robust inference matched to the actual randomization level.

Part 6 — Hands-on A/B and A/A test

You are given a reservation-level experiment table with these columns:

Hint — A/B: Filter to is_aa_test == False, validate one row per unit_id and that assigned_at precedes outcomes, then run the test on support_contacted (or your chosen primary). used_locker is post-treatment — never condition the main estimate on it. Hint — A/A: The A/A rows (is_aa_test == True) should show no real effect. Two checks: a sample-ratio-mismatch test (chi-square of variant counts vs. expected split) and covariate balance on pre_assignment_trips / pre_assignment_support_rate / market. In large samples, prefer standardized mean differences over raw p-values for balance — tiny imbalances become 'significant.'

What This Part Should Cover

• A column-specific, runnable A/B procedure: filter on is_aa_test, validate unit_id uniqueness and that assigned_at precedes outcomes, estimate ITT on the chosen primary.
• Treats used_locker strictly as post-treatment (a diagnostic, never a filter or conditioning variable for the headline).
• An A/A that runs the SRM (chi-square) check and covariate balance via standardized mean differences, and explains what a failure implies for trusting the live A/B.

Part 7 — Risks

Discuss the risks that threaten this specific experiment: selection bias, interference between treatment and control, locker capacity constraints, host-side spillovers, and heterogeneous effects across markets. For each, name how your design or analysis mitigates it.

Hint: Each risk maps to a design lever you already chose: selection bias → ITT; interference/capacity → choice of randomization unit (cluster) + capacity-aware analysis; host spillovers → listing/host-level unit; heterogeneity → pre-specified market stratification (not subgroup fishing).

What This Part Should Cover

• A risk-to-mitigation mapping (each risk paired with the specific design or analysis lever), not a generic risk list.
• Recognizes that the mitigations are the same levers chosen earlier (unit choice, ITT, stratification), showing the design hangs together.
• Calls out heterogeneity discipline — pre-specified stratification, not post-hoc subgroup fishing.

What a Strong Answer Covers

These dimensions span all parts and tie the case together:

• Separates the three problems: the product decision, the randomization design, and the statistical analysis — and does not conflate 'guests like it' with 'the marketplace is better off.'
• Treats the two-sided marketplace as a first-class constraint throughout (guest wins are gated by host, safety, cost, and capacity guardrails).
• Consistency across parts: the randomization unit, estimand, metrics, power math, and risk mitigations all reference each other rather than contradicting.
• Pre-registration discipline: MDE, primary metric, and any subgroup analyses are fixed before looking at data.

Follow-up Questions

• The treatment effect on the primary metric is positive and significant, but locker over-capacity rate breaches its guardrail in two high-volume markets. What do you recommend, and what is the next experiment?
• The A/A test shows a 2% sample-ratio mismatch (e.g., 51/49 instead of 50/50) that is statistically significant. Walk through how you would diagnose whether it is a logging artifact or a real assignment bug, and whether you would trust the A/B result.
• How would you estimate the treatment-on-treated (TOT) effect for guests who actually used a locker, and what assumption does using assignment as an instrument require?
• Suppose the effect is strongly heterogeneous — large in dense urban markets, negative in rural ones. How does that change the launch decision and the rollout plan? Overview: This question assesses a candidate's skills in experimental design and causal inference for marketplace features, including selecting the appropriate randomization unit, defining eligibility and treatment/control arms, and choosing primary, secondary, and guardrail metrics.