Choose a single skill emphasized in the job description for this position and one resume project where you used it. Cover: (1) the problem, constraints, and success metric; (2) the precise techniques and tools (versions, scale, non-obvious design decisions) you applied; (3) one meaningful failure or edge case and how you fixed it; (4) quantified before/after impact and what you would change on a second attempt; (5) how you reduced risk with stakeholders and the trade-offs you deliberately accepted.

Overview: The question tests whether you can connect a particular skill from the job description to a prior project, showing technical data science ability, quantified impact, and stakeholder communication.

Solution

Sample answer intended for teaching

Skill chosen from the job description: Experimentation and causal inference (A/B testing, metric design). Resume project: A personalization experiment aimed at improving homepage ranking at a large subscription streaming service.

1) Problem, constraints, and success metric

• Problem: Boost content discovery from the homepage without degrading quality of experience (QoE).
• Constraints:
  • Latency: p95 homepage render plus ranking budget under 150 ms.
  • Global rollout across regions, languages, and device types (TV, mobile, web).
  • Guardrails: No material increase in rebuffering or error rates; no policy or rights violations.
  • Experiment overlap policy: Mutually exclusive buckets with other homepage tests.
• Primary success metric:
  • 7-day Play Starts per Profile (PSP). Secondary: 7-day Watch Time per Profile (minutes). Guardrails: start-failure rate, rebuffering ratio, crash rate.
• MDE/power target:
  • Detect a 1.0% relative lift in PSP with 80% power and $\alpha = 0.05$.
  • Two-sample per-arm size approximation: $n \approx 2 \cdot (z_{1-\alpha/2}+z_{1-\beta})^2 \cdot \sigma^2 / \delta^2$.
    • Example inputs: baseline mean $\mu=2.9$ plays, $\sigma=3.2$, $\delta=0.029$ (1% of $\mu$), $z_{0.975}=1.96$, $z_{0.8}=0.84$.
    • $n \approx 2 \cdot (1.96+0.84)^2 \cdot (3.2^2)/(0.029^2) \approx 191k$ profiles/arm (before variance reduction). We expected to reach this in under one day.

2) Techniques, tools, and non-obvious design decisions

• Experiment design:
  • Randomization unit: profile level; cluster-robust analysis at the household level to reduce cross-device interference.
  • 50/50 allocation, stratified by region × device to improve balance and power.
  • Variance reduction: CUPED with 14-day pre-experiment covariates (prior play starts, watch time, tenure, device).
    • CUPED formula: $Y_{adj} = Y - \theta \cdot (X - E[X])$, where $\theta = \mathrm{Cov}(Y, X) / \mathrm{Var}(X)$.
• Analysis:
  • Primary estimator: difference-in-means on per-profile outcomes; confirmatory OLS with covariates and cluster-robust standard errors (household clustering).
  • Ratio metrics handled through per-profile aggregation (avoiding per-event ratios) and delta-method checks; confirmed with a nonparametric bootstrap (10k replicates).
  • Sequential monitoring with alpha-spending (Pocock boundary) to prevent inflated Type I error during gated ramps.
• Ranking/modeling:
  • Offline reranking blend: baseline collaborative filtering plus short-term session signals; restricted to top-N candidates to remain within latency.
  • Non-obvious choice: Winsorized extreme watch time at 99.5% to stabilize variance; capped per-request reranking at 50 candidates to fit p95 latency.
• Tooling and scale:
  • Data/compute: PySpark 3.3 on Spark 3.3 (Databricks Runtime 12.x), Delta tables.
  • Orchestration: Airflow 2.6 for daily ETL and metric rollups; MLflow 2.6 for experiment metadata.
  • Stats: Python 3.10, statsmodels 0.14, SciPy 1.10; visualization in a BI tool for stakeholder readouts.
  • Scale: about 12M profiles in the experiment over 14 days; roughly 2B events per day feeding metrics.

3) Nontrivial failure/edge case and resolution

• Issue: Sample Ratio Mismatch (SRM) on Android WebView (51.3/48.7 split, $p < 10^{-4}$). Root cause was CDN-level caching of the pre-assigned homepage for some anonymous sessions before server-side assignment was finalized.
• Resolution:
  • Moved assignment earlier in the server-side request pipeline; used a stable profile_id-based Murmur3 hash for bucketing.
  • Added real-time SRM monitoring (hourly Pearson $\chi^2$ across key strata) and blocked enrollment when SRM triggered.
  • After the fix, arm proportions were within ±0.1% of expected across strata; we invalidated pre-fix data and restarted the experiment.
• Lesson: For pages served behind aggressive edge caches, make sure treatment assignment happens upstream of any cacheable content and that anonymous flows receive a stable assignment key.

4) Impact (before/after) and what I would change next time

• Results (14 days, after fix, CUPED-adjusted):
  • +1.8% relative lift in 7-day Play Starts per Profile (ATE +0.052 from a 2.90 baseline), 95% CI [+1.0%, +2.6%], $p=0.001$.
  • +1.2% lift in 7-day Watch Time per Profile (about +4.1 minutes), 95% CI [+1.4, +6.8] minutes.
  • Guardrails: Rebuffering +0.03pp (ns), start-failure −0.02pp (ns). No material QoE regressions.
  • Heterogeneity: Larger lift for new users (<30 days tenure): +3.4% PSP; stable for long-tenure users.
• Business translation:
  • At full rollout scale, the lift implies several million additional weekly play starts with stable QoE.
• If doing it again:
  • Pre-register stratified MDEs and power by user tenure to right-size ramp windows.
  • Add CUPAC (covariate-assisted randomization) to further reduce variance and speed decisions.
  • Use a short pre-launch shadow test with off-policy evaluation (doubly robust estimator) to catch SRM-like issues before the live ramp.

5) De-risking with stakeholders and conscious trade-offs

• De-risking steps:
  • Alignment on primary and guardrail metrics and decision thresholds before launch; documented in a one-pager and pre-registered.
  • Gated rollout: 1% → 5% → 20% → 50% with alpha-spent interim looks and automatic rollback on QoE guardrail breaches.
  • Data-quality checks in Airflow using Great Expectations (schema, nulls, range checks) and automated SRM alerts.
  • Mutually exclusive bucketing with other homepage experiments to avoid interference.
• Trade-offs chosen:
  • Interpretability over speed: kept a 50/50 RCT rather than a bandit to obtain clean ATEs and learn across segments; accepted slightly slower convergence.
  • Latency budget over model complexity: bounded reranking candidates and used lightweight features; deferred heavier context features to a follow-up.
  • Variance reduction (CUPED, stratification) over longer runtime: invested upfront in design to hit MDE sooner without over-ramping.

Why this maps to the JD skill: The project demonstrates end-to-end experimentation rigor—powering, randomization strategy, variance reduction, SRM detection, robust inference, guardrail governance—and turns results into product decisions with quantified impact and clear trade-offs.