Roblox · Statistics & Data Analysis

Design ad revenue A/B with guardrails

TrueInterview

October 7, 2026 · 1 min read

You operate a mobile app advertising marketplace. The objective is to raise ad revenue by 10% within 60 days without allowing 7-day retention to decline by more than 0.5 percentage points or average session length to decline by more than 2%. Baseline figures: DAU is 1,000,000; users average 3 sessions per day; each session averages 2 impressions; CTR is 2%; CPM is $5; fill rate is 90%; and per-user daily revenue variance is roughly 0.04 (USD^2). Propose an experiment plan to test two levers: (A) a 10% increase in ad load; (B) a $0.10 increase in floor price. Answer:

Pick a randomization unit—user, session, or geo—and justify the interference risk posed by auction dynamics and supply constraints. How will you reduce cross-treatment spillover? Would you favor cluster randomization, ghost ads, or geo holdouts?
Define the primary KPI(s) used for decisioning, such as revenue per user per day, and the guardrails for 7-day retention, session length, and crash rate. Specify exact success and stop-loss thresholds.
For the power analysis, compute the required sample size, traffic split, and duration for 90% power at $\alpha=0.05$ to detect a 10% uplift in revenue per user per day. State your assumptions—variance, and ICC if clustering is used—and show the formulas.
For bias control, outline how you would apply CUPED or pre-experiment covariates, control for seasonality, and decide between sequential and fixed-horizon testing. Provide your decision rule and how you will adjust for multiple variants.
For advertiser effects, how will you detect or avoid cannibalization between campaigns and shifts in clearing prices? What diagnostics would you run, such as win rate, eCPM distribution, or supply-demand curves?

Overview: This question assesses a data scientist's abilities in experimental design, causal inference, and statistical power analysis for ad monetization, including randomization unit choice, interference mitigation in auctioned inventory, guardrail definition, bias controls, and advertiser-effect diagnostics.

Roblox · Statistics & Data Analysis

Design ad revenue A/B with guardrails

TrueInterview

October 7, 2026 · 1 min read

Pick a randomization unit—user, session, or geo—and justify the interference risk posed by auction dynamics and supply constraints. How will you reduce cross-treatment spillover? Would you favor cluster randomization, ghost ads, or geo holdouts?
Define the primary KPI(s) used for decisioning, such as revenue per user per day, and the guardrails for 7-day retention, session length, and crash rate. Specify exact success and stop-loss thresholds.
For the power analysis, compute the required sample size, traffic split, and duration for 90% power at $\alpha=0.05$ to detect a 10% uplift in revenue per user per day. State your assumptions—variance, and ICC if clustering is used—and show the formulas.
For bias control, outline how you would apply CUPED or pre-experiment covariates, control for seasonality, and decide between sequential and fixed-horizon testing. Provide your decision rule and how you will adjust for multiple variants.
For advertiser effects, how will you detect or avoid cannibalization between campaigns and shifts in clearing prices? What diagnostics would you run, such as win rate, eCPM distribution, or supply-demand curves?