Pick one consumer mobile app you personally open every week. 1) Put forward exactly six concrete improvements that could realistically ship; for each, name the target user, the user behavior you want to shift, the primary success metric plus a 14-day leading indicator, and the biggest execution risk. 2) Sketch a rough back-of-the-envelope model estimating each idea's 90-day effect on the app's north-star metric; state every assumption and give low/base/high ranges. 3) Rank the six with a transparent framework (RICE or expected ROI, for example) under the constraint of one squad for one quarter; name the top priority and justify the trade-offs you accept as CEO (revenue vs. retention, complexity, brand risk, and so on). 4) Define an experiment plan for the winning idea: experiment design, go/no-go criteria, guardrail metrics, and explicit kill conditions if early signals disappoint. 5) Name two non-obvious failure modes and how you would mitigate them before and after launch.

Overview: This question assesses a data scientist's product thinking, prioritization, impact estimation, experiment design and risk-mitigation skills, framed in the Behavioral & Leadership domain and centered on product management, modeling and experimentation.

Solution

App and North-Star Metric

App chosen: Spotify (consumer music and podcast streaming). North-star metric (NSM): Weekly Listening Minutes (WLM) across the target market. It captures user value (time spent listening), moves with both retention and revenue (advertising and premium satisfaction), and can be measured on a weekly cadence.

Baseline assumptions (modeling a single large market/region):

• Weekly Active Users (WAU): 10,000,000
• Baseline sessions per WAU per week: 6
• Baseline average minutes per session: 25
• Baseline WLM per week = WAU × sessions × minutes = 10M × 6 × 25 = 1,500M minutes

$\text{WLM}_{\text{week}} = \text{WAU} \times \text{sessions} \times \text{minutes} = 10M \times 6 \times 25 = 1{,}500M$

• 90 days ≈ 13 weeks

All impact estimates include adoption ramp factors and low/base/high ranges.

1) Six Concrete, Shippable Improvements

1. Contextual "Smart Start" Quick Play

• Target user: Light to medium listeners who browse Home before playing.
• Behavior change: Cut time-to-first-play and raise weekly starts.
• Primary success metric: Sessions that begin playback within 15 seconds of app open; WLM uplift among eligibles.
• 14-day leading indicator: "Play within 15s" up 2–4 percentage points, plus 0.1–0.3 additional sessions/week among impacted users.
• Largest execution risk: Mis-targeting damages trust when suggestions feel off; model complexity could slow cold start.

2. One-Tap "Micro-Playlist" Builder (10-song mood list)

• Target user: Semi-engaged users who enjoy curation but seldom build playlists.
• Behavior change: Raise the share of listening that comes from user-created lists and repeat plays.
• Primary success metric: % of listening minutes from user-created playlists; WLM per impacted user.
• 14-day leading indicator: % of eligibles who create 1 micro-playlist; 7-day repeat play rate for those lists.
• Largest execution risk: Adoption stays low if creation still feels like work; Home becomes cluttered.

3. Podcast-to-Music "Blend"

• Target user: Users who finish a podcast and then leave the app.
• Behavior change: Keep the session going by moving into relevant music.
• Primary success metric: Post-podcast continuation rate; WLM from post-podcast sessions.
• 14-day leading indicator: Transitions per WAU; minutes played in the first 10 minutes after a podcast ends.
• Largest execution risk: Annoyance when users expect quiet after podcasts; recommendations that miss the mark.

4. Weekly Listening Goals + Gentle Streaks

• Target user: Light users whose weekly engagement is inconsistent.
• Behavior change: Lift weekly sessions and cut short-term churn.
• Primary success metric: 7/28-day retention in the lowest engagement cohorts; WLM per impacted user.
• 14-day leading indicator: Goal opt-in rate; streak completion rate; 0.15–0.45 extra sessions/week among impacted users.
• Largest execution risk: Backlash against gamification, feeling manipulated; perverse incentives such as short sessions kept alive just to hold a streak.

5. Smart Offline "Download Next"

• Target user: Commuters and users on patchy connectivity who often cannot listen because there is no service.
• Behavior change: Grow offline listening minutes by auto-downloading the next episodes/tracks while on Wi-Fi and charging.
• Primary success metric: Offline WLM per impacted user.
• 14-day leading indicator: Opt-in rate; % of offline sessions with zero playback errors.
• Largest execution risk: Storage and data-usage worries; unexpected downloads that damage trust.

6. Lyrics "Tap-to-Clip & Share" (auto-captioned 10–20s snippets)

• Target user: Users who read lyrics and share music socially.
• Behavior change: Drive re-entries and pull friends in through social shares.
• Primary success metric: WLM attributable to share creators and recipients (openers).
• 14-day leading indicator: Share creation rate; share open rate; minutes per referred session.
• Largest execution risk: Licensing/UGC brand risk (copyrighted lyrics, offensive content) and platform policy friction.

2) Back-of-the-Envelope Impact Modeling (90 Days)

Notation:

• Impacted users = WAU × eligibility × adoption

$\text{Impacted users} = \text{WAU} \times \text{eligibility} \times \text{adoption}$

• Per-user weekly delta minutes = (Δsessions × baseline minutes/session) + (baseline sessions × Δminutes/session), unless stated otherwise

$\Delta_{\text{user/week}} = (\Delta \text{sessions} \times \text{baseline minutes/session}) + (\text{baseline sessions} \times \Delta \text{minutes/session})$

• 90-day incremental WLM ≈ weekly delta × 13 × ramp factor

$\text{90-day incremental WLM} \approx \text{weekly delta} \times 13 \times \text{ramp factor}$

Global baselines: WAU=10M, sessions=6, minutes/session=25, baseline WLM/week=1,500M.

1. Smart Start Quick Play

• Impacted users: 5M (50% of WAU see the surface and are influenced)
• Assumptions (low/base/high):
  • Δsessions/week: 0.1 / 0.2 / 0.3
  • Δminutes/session: 0.4 / 1.0 / 1.5
  • Ramp factor over 90d: 0.6 / 0.7 / 0.8
• Per-user weekly delta (base): 0.2×25 + 6×1.0 = 11.0 minutes
• Weekly total delta (base): 11.0 × 5M = 55.0M minutes
• 90d incremental WLM: 55.0×13×0.7 = 500.5M Range: 190.5M (low) to 858.0M (high)

2. Micro-Playlist Builder

• Impacted users: 1M (40% eligible × 25% adoption ≈ 10% of WAU)
• Assumptions:
  • Δsessions/week: 0.02 / 0.05 / 0.08
  • Δminutes/session: 0.5 / 1.5 / 2.5
  • Ramp: 0.5 / 0.6 / 0.7
• Per-user weekly delta (base): 0.05×25 + 6×1.5 = 1.25 + 9 = 10.25
• Weekly total (base): 10.25M
• 90d incremental: 10.25×13×0.6 = 79.9M Range: 22.8M to 154.7M

3. Podcast-to-Music Blend

• Impacted users: 1M (20% podcast listeners × 50% impacted)
• Assumptions:
  • Extra minutes per week post-podcast: 3 / 8 / 12
  • Ramp: 0.6 / 0.7 / 0.8
• Weekly total (base): 8M
• 90d incremental: 8×13×0.7 = 72.8M Range: 23.4M to 124.8M

4. Weekly Goals + Streaks

• Impacted users: 1.5M (bottom 50% WAU × 30% opt-in)
• Assumptions:
  • Δsessions/week: 0.15 / 0.30 / 0.45
  • Δminutes/session: 0.0 / 0.5 / 1.0
  • Ramp: 0.5 / 0.6 / 0.7
• Per-user weekly delta (base): 0.30×25 + 6×0.5 = 7.5 + 3 = 10.5
• Weekly total (base): 10.5×1.5M = 15.75M
• 90d incremental: 15.75×13×0.6 = 122.9M Range: 36.6M to 235.4M

5. Smart Offline Download Next

• Impacted users: 0.6M (15% WAU × 40% opt-in)
• Assumptions:
  • Extra minutes/week: 5 / 12 / 20
  • Ramp: 0.6 / 0.7 / 0.8
• Weekly total (base): 0.6M×12 = 7.2M
• 90d incremental: 7.2×13×0.7 = 65.5M Range: 23.4M to 124.8M

6. Lyrics Tap-to-Clip & Share

• Impacted creators: 0.6M (30% lyrics viewers × 20% share)
• Assumptions:
  • Creator extra minutes/week: 1 / 2.5 / 5
  • Recipient minutes/week (aggregate): 0.5M / 1.5M / 3.0M
  • Ramp: 0.4 / 0.5 / 0.6
• Weekly total (base): (0.6M×2.5) + 1.5M = 3.0M
• 90d incremental: 3.0×13×0.5 = 19.5M Range: 5.7M to 46.8M

Notes and pitfalls:

• Effects are assumed independent; in practice some ideas cannibalize others (Smart Start vs. Streaks, for instance). Do not double-count when planning the portfolio.
• Ramps capture engineering rollout, discovery and habit formation; if you hard-gate by platform, lower the ramp.
• For high-variance segments (heavy listeners), stratify in the experiments to cut noise.

3) Prioritization Under One-Squad/One-Quarter Constraint

Framework: Expected ROI = 90-day incremental WLM (base case) per squad-week of effort. Effort is estimated inclusive of design, eng, ML, QA and data work.

$\text{Expected ROI} = \frac{\text{90-day incremental WLM (base case)}}{\text{squad-weeks of effort}}$

Effort estimates (squad-weeks):

• Smart Start: 10 (ranking changes, caching, telemetry, UX)
• Micro-Playlist: 6 (light backend + UI)
• Podcast-to-Music: 5 (eligibility, recommender hook, UX)
• Streaks: 7 (state machine, notifications, abuse safeguards)
• Offline Next: 8 (download scheduler, storage, settings)
• Lyrics Share: 9 (editorial filters, export, rights review)

ROI (base 90d WLM / effort):

• Smart Start: 500.5 / 10 = 50.1
• Streaks: 122.9 / 7 = 17.6
• Podcast-to-Music: 72.8 / 5 = 14.6
• Micro-Playlist: 79.9 / 6 = 13.7
• Offline Next: 65.5 / 8 = 8.2
• Lyrics Share: 19.5 / 9 = 2.2

Priority order:

1. Smart Start
2. Streaks
3. Podcast-to-Music
4. Micro-Playlist
5. Offline Next
6. Lyrics Share

Top pick: Smart Start. Why: Best expected ROI, wide reach, direct tie to the NSM, and lower brand/licensing risk than social sharing. It strengthens core listening behavior instead of vanity metrics. Trade-offs accepted: giving up virality (Lyrics Share) and narrow segment wins (Offline) to fund a platform-level, always-on improvement. Complexity is manageable in one quarter with a single squad if you scope to Home and Launch surfaces first.

4) Experiment Plan for Smart Start

Experiment design

• Unit: User-level randomized A/B test among eligible users (iOS/Android), 50/50.
• Duration: 2–3 weeks for leading indicators; continue 4–6 weeks on a subset to validate WLM and early retention effects.
• Segmentation: Pre-stratify by engagement tier (light/medium/heavy), platform and country. Keep cohorts equally represented.
• Sample size: Target detecting a +1.0% lift in WLM among eligibles. Approximation:
  • Baseline weekly minutes per WAU ≈ 150; stdev ≈ 200
  • δ = 1.5 minutes; α = 0.05; power = 0.8

$\delta = 1.5 \text{ minutes},\quad \alpha = 0.05,\quad \text{power} = 0.8$

$n \approx 2 \times (Z_{\alpha/2} + Z_{\beta})^2 \times \sigma^2 / \delta^2 \approx 2 \times (2.8)^2 \times 40{,}000 / 2.25 \approx 280k \text{ users/arm}$

Round to 300–500k/arm.

Metrics

• Primary: Weekly Listening Minutes among eligibles; Sessions starting playback within 15s of open.
• Secondary: Sessions per WAU; Avg minutes per session; % of play starts from Smart Start.
• Guardrails:
  • 7-day retention (no worse than −0.1pp vs. control overall and by cohort)
  • Skip rate per hour (+ ≤2% absolute vs. control)
  • Crash-free sessions (no degradation >0.1pp)
  • App cold-start time (+ ≤50ms)
  • Support tickets/negative feedback mentioning recommendations (no >10% increase)
  • Battery/data usage on foreground launch (no >3% increase)

Go/No-Go criteria

• Go if BOTH hold (with 95% CI lower bound > 0):
  • +1.0% or more lift in WLM among eligibles by week 2–3
  • +2pp or more increase in "play within 15s" by day 14
• And ALL guardrails stay within thresholds.

Kill/rollback conditions (early)

• By day 7: "play within 15s" lift < +0.5pp AND skip rate worsens >2% absolute.
• By day 14: 95% CI for WLM uplift includes 0 AND at least one guardrail breached for 3 consecutive days.
• Any P0 privacy/security incident or >0.2pp decline in 7-day retention in any major cohort.

Ramp plan

• Phase 0: Internal dogfood + 1% external traffic (1 week).
• Phase 1: 10% traffic with A/A holdout to validate instrumentation.
• Phase 2: 50% A/B with cohort stratification.
• Phase 3: If Go, ramp to 100% over 1–2 weeks with a long-term holdout (5%) to watch for recommender feedback loops.

Implementation guardrails

• Remote-config toggle; safe fallback to existing Home modules.
• Caching to avoid cold-start latency; precompute candidates daily, rerank on launch.

5) Two Non-Obvious Failure Modes and Mitigations (Smart Start)

1. Recommender feedback loop narrows catalog diversity over time

• Risk: Over-optimizing launch suggestions toward the same high-CTR tracks homogenizes listening, eroding long-term satisfaction and artist diversity.
• Pre-launch mitigation: Impose diversity constraints (cap the share from the top decile of popularity, keep genre/artist variety). Reserve exploration buckets (~5–10%) in ranking.
• Post-launch mitigation: Track weekly unique artists per user, long-tail share of minutes and new-artist discovery rate. If diversity falls >5–10% vs. control, raise exploration weight or add novelty boosts.

2. Context signals quietly encode sensitive attributes, creating privacy/regulatory risk

• Risk: Using location/time/activity to infer context can correlate with sensitive traits (places of worship, medical facilities, and the like), producing privacy exposure and reputational damage.
• Pre-launch mitigation: Data minimization (coarse time-of-day and device state only), a privacy impact assessment, on-device processing where possible, and an explicit opt-out. Drop precise location and never combine it with sensitive categories.
• Post-launch mitigation: Logging review for sensitive fields, automated scans for policy violations, and a fast rollback path if regulators or users raise concerns.

Closing Notes

• The plan favors immediate, broad improvements to core listening behavior (Smart Start) over narrower or riskier bets (social sharing, heavy offline). If early results fall short, the next-highest ROI options (Streaks, Podcast-to-Music) offer diversified paths — one retention-leaning, one session-continuity — while keeping the NSM focus intact.