You have one hour with a supplied dataset — no chance to look at it in advance — and then a 45-minute debrief with a product analyst and stakeholders. Lay out exactly how you would: (1) triage the data and land on a decision-oriented objective inside the first 10 minutes; (2) pick 3–5 core metrics plus a small set of slices to separate signal from noise; (3) shape a 5-slide narrative (title, problem, method, results, risks/assumptions, decision/next steps) that non-technical stakeholders can follow; (4) convey uncertainty and caveats without eroding confidence; (5) deal with an insistent stakeholder who demands a conclusion the data cannot support; (6) write out 2–3 specific phrases you would use to steer the discussion and negotiate the scope of follow-up work.

Overview: This prompt tests a data scientist's ability to triage analysis at speed, select metrics aimed at a decision, tell a tight story to a non-technical audience, communicate uncertainty, and lead through pushback. It is categorized under Behavioral & Leadership within Data Science.

Read the complete Thumbtack Data Scientist interview account this question came from

Solution

Overview

Goal: within 60 minutes, build a defensible narrative that is ready to drive a decision, then spend 45 minutes walking through it. Assume an ordinary product dataset (events or transactions) carrying timestamps and user/item identifiers. The plan below is time-boxed, decision-first, and tolerant of incomplete metadata.

1) First 10 minutes: triage + decision-focused objective

Rough time-boxed checklist:

• 0–2 minutes: Frame the decision
  • Ask or work out: "If the data settled it, what decision would we make today?" Examples: ship or roll back a feature, prioritize a defect, aim at a segment, launch an experiment.
  • Write a one-sentence objective you can read aloud in the debrief: "Estimate the impact of X on Y and recommend the next decision (ship, rollback, test, or instrument)."
• 2–7 minutes: Data triage
  • Pin down the grain: does one row mean a user, a session, an event, or an order?
  • Read the schema: enumerate columns and data types, spot the obvious keys (user_id, item_id, timestamp), and note feature flags or experiment arms.
  • Fast sanity checks:
    • Row counts and date range: earliest and latest timestamp, plus volume per day to reveal outages.
    • Uniqueness: verify the primary key is unique; dedupe where needed.
    • Missingness: percent null by column; drop or flag columns you cannot use.
    • Reasonableness: negative prices, implausible ages, timestamp anomalies.
  • When experiment columns are present: look for sample ratio mismatch (SRM) in the variant counts.
• 7–10 minutes: Lock the decision-focused objective
  • Turn the triage into a decision statement with success criteria: "We will estimate the directional change in primary metric Y against baseline, surface 1–2 high-variance slices, and recommend an A/B test or a rollout if the 95% CI excludes zero and guardrails hold."
  • Record assumptions: the data spans the last N weeks, and no major confounders beyond [platform, new vs. returning, geo]. Deliverables at minute 10:
• A one-sentence decision objective
• Draft metric definitions
• A short list of slices to inspect

2) Core metrics (3–5) and minimal slices

Choose metrics tied straight to the decision, spanning value, volume, quality, and speed. Examples with formulas:

• Primary outcome (conversion):
  • Conversion rate (CR): $\text{CR} = \frac{\text{success events}}{\text{eligible users or sessions}}$
• Value: revenue per active user (RPU): $\text{RPU} = \frac{\text{total revenue}}{\text{active users}}$ — or GMV per requester.
• Quality: defect/cancellation rate: $\frac{\text{defective events}}{\text{completed events}}$
• Engagement/throughput: request rate: $\frac{\text{requests}}{\text{active user}}$ — or quotes per request.
• Speed-to-value: median time-to-first-success (time-to-first-quote or time-to-booking, for example).

Minimal slice set to separate signal from noise (keep to 3–5):

• Time: day or week (to catch outages and trends).
• Platform: iOS vs Android vs Web (a common source of heterogeneity).
• Cohort: new vs returning users (behavior differs materially).
• Channel/geo: the top 2–3 acquisition channels or regions by volume.
• Experiment/feature flag: treatment vs control (where one exists).

How to choose slices fast:

• Lean on Pareto: take the 3–4 dimensions that account for at least 80% of volume.
• Hold a minimum sample size per slice (500–1,000 observations for proportions, scaled to the expected effect). Merge or drop anything below the bar.

Quick signal vs noise check for proportions:

• For a proportion $p$ over $n$ trials, the standard error is roughly $SE \approx \sqrt{\frac{p(1-p)}{n}}$, and a ballpark 95% CI is $p \pm 1.96 \times SE$.
• Example: baseline CR = 12% with $n = 10000$ gives $SE \approx \sqrt{\frac{0.12 \times 0.88}{10000}} \approx 0.0032$ (0.32pp), so the 95% CI is about $12\% \pm 0.63\text{pp}$. A segment sitting at 8% (a 4pp gap) is far outside noise.

Pitfalls to avoid:

• Ratio of means vs mean of ratios: set the denominator at the decision unit (usually the user).
• Simpson's paradox: compare the overall figure with within-key slices (platform, for instance).
• Multiple comparisons: treat wide slice fishing as hypothesis generation and confirm it later.

3) Five-slide narrative for non-technical stakeholders

Hold it to 5 slides by folding risks/assumptions in with decision/next steps.

• Slide 1 — Title
  • Title plus a one-sentence objective
  • Data window, dataset(s), and unit of analysis
• Slide 2 — Problem & Decision
  • The business question and why it matters (impact proxy: users, revenue, quality)
  • The options on the table: ship, rollback, iterate, A/B test, instrument
  • Success threshold (pre-committed where possible): we need at least a +2% CR lift with guardrails steady
• Slide 3 — Method (plain language)
  • Definitions of the 3–5 metrics; unit and denominator
  • Which slices were examined and why (platform, cohort, time)
  • Data checks: coverage dates, missingness, SRM
  • Analytical approach: straightforward comparisons with CIs, plus trend and segment cuts
• Slide 4 — Results
  • 1–2 clean visuals with effect sizes and 95% CIs by key slice
  • Callouts: the top 2 insights, their magnitude, and the segments at risk
  • A one-sentence takeaway under each chart
• Slide 5 — Risks, Assumptions, Decision & Next Steps
  • Assumptions: representativeness, attribution caveats
  • Risks: data gaps, confounders, small-n segments
  • Decision: the recommended action and why
  • Next steps: confirmatory test or instrumentation, timeline, owner

4) Communicating uncertainty without undermining confidence

Principles and phrasing:

• Open with the decision, then give the range: "We recommend an A/B test; estimated impact is +2–4% CR with quality holding steady."
• Express uncertainty in units people feel: "±0.6 percentage points" or "roughly 20–40 extra conversions per 1,000 users."
• Keep two kinds of uncertainty apart:
  • Statistical: CIs/SEs, sample sizes
  • Data-quality: coverage gaps, missingness, SRM
• Work with thresholds and a stoplight frame:
  • Green: the CI sits fully above the threshold and guardrails are stable
  • Yellow: the direction is positive but the CI overlaps; propose a test or monitoring
  • Red: the CI straddles zero or a guardrail is breached; do not ship
• Head off overreach: "These are associations, not causal effects, absent randomization."

If experimentation is involved:

• Validate randomization: check sample ratio mismatch (chi-square on variant counts).
• Guardrails: bounce rate, cancellations, support tickets; require no material degradation.
• Power check: if underpowered, recommend extending duration or increasing sample.

5) Handling a pushy stakeholder insisting on an unsupported conclusion

Playbook:

• Recognize the intent behind it: "I understand the urgency to ship."
• Return to the thresholds and data limits you agreed on: "Our CI overlaps zero and the guardrails are still uncertain."
• Offer a path with contained risk: "We could ship behind a flag at 5% and watch the guardrails," or "Run a one-week A/B with explicit stop criteria."
• Escalate to the decision framework: "Given the downside risk of X and where we stand on certainty, an experiment buys the most learning per unit time."
• Document: capture the disagreement, the decision criteria, and the next steps in the recap email.

6) Scripted phrases to redirect and negotiate scope

Use short, respectful, repeatable language.

• Redirect to decision and evidence: "For a high-confidence call today, what matters is whether the 95% interval clears our +2% threshold without hurting cancellations. It doesn't right now, so the lowest-risk move is a short A/B test."
• Name the uncertainty and offer a path: "The data points in a positive direction, but the CI still crosses zero. Put 10% of traffic behind it for a week and we'll have the power to confirm or pivot."
• Constrain scope and align on follow-ups: "With the time we have, I can cover the primary question and one deep-dive slice. For the rest, I'd suggest a 24–48 hour follow-up with instrumentation notes and a test plan. Does that work for you?"

---

Appendix: quick calculation guardrails (use as needed)

• Proportion CI: $p \pm 1.96\sqrt{\frac{p(1-p)}{n}}$
• Difference in proportions (A vs B) SE: $SE \approx \sqrt{\frac{p_A(1-p_A)}{n_A} + \frac{p_B(1-p_B)}{n_B}}$
• Small samples: use Wilson interval or bootstrap
• Continuous metrics: report medians or trimmed means if heavy tails
• Multiple slices: treat as exploratory; confirm with targeted tests