Analyze A/B test with revenue–cost tradeoffs

You are given a take-home A/B test concerning a checkout modification that advertises same-day delivery. The main metrics are: conversion rate, average order value (AOV), take rate (the platform's percentage of GMV), and delivery cost per order. Data properties: roughly 50,000 users per arm ($n \approx 50k$); AOV and cost are right-skewed; variances differ between arms ($\text{variance ratio} \approx 2$); 15% of orders incur no delivery cost because of promotions. a) Select and defend the appropriate hypothesis test(s) for each metric (for instance, Welch's t-test versus standard t-test versus nonparametric or bootstrap methods versus z-test), and state the distributional assumptions you depend on and your approach to checking them. b) Measure the net revenue effect: define and calculate contribution per order as $\text{GMV} \times \text{take\_rate} - \text{delivery\_cost}$, permitting delivery_cost to rise with order volume and recognizing that take_rate plateaus under large discounts. Demonstrate how you would model cost scaling (such as a piecewise or queueing-based function) and the diminishing take_rate, and how you would carry uncertainty through to a final decision. c) Suggest a segmentation strategy (for example, by market, daypart, basket size, new versus repeat customers) to uncover heterogeneous treatment effects; describe how you would manage multiplicity and prevent overfitting while still identifying practical segments. d) Considering the operational risk of manually boosting shopper supply, propose an optimization objective and constraints that trade off short-term contribution gains against service-level metrics (SLAs, cancellation risk).

Overview: This item gauges a data scientist's proficiency in designing experiments and drawing statistical inferences for A/B tests, modeling revenue impact when metrics are skewed and zero-inflated, segmenting to find heterogeneous treatment effects, and solving constrained optimization problems that balance contribution against operational risk.