Define success metrics and guardrails for B2B chat

A rigorous plan for measuring success must be defined for the new EU business-to-customer chat subscription. Your specification must include: (a) five primary KPIs that capture early product-market fit and monetization (for example, Number of Companies Subscribing Monthly, Revenue from Monthly Subscriptions, Proportion Resolved Within 24 Hours, First Response Time at 50th and 90th Percentiles, Net Revenue Retention); (b) six guardrails that prevent customer harm or spam (for example, Percentage of Chats Not Solved, Rate of Long-Duration Chats, Rate of Unexpected Message Bursts, Complaint Rate, Unsubscribe/Opt-out Rate, Share of Negative Sentiment); and (c) precise operational definitions including formulas, event filters, and attribution rules (multi-language, bot versus human handoff, multi-agent threads, reopened tickets). For 'Not Solved,' supply two measurable definitions: one that relies on a threshold from a sentiment model and one that relies on CSAT collected after the chat; for each, discuss biases and failure modes and how you would calibrate the thresholds. Also incorporate temporal windows (e.g., month-over-month, 28-day rolling), cohort analysis (by month the business signed up and by month of the customer's first contact), segmentation (country, vertical, company size), and data quality checks (outliers, duplicate threads, late events). For every KPI and guardrail, propose target ranges and escalation triggers.

Overview: This question assesses skills in analytics and experimentation, design of product metrics, attribution, and operational measurement—including key performance indicators, guardrails, event filters, cohorting, segmentation, calibration, and data quality checks for a paid EU B2C chat subscription within the Analytics & Experimentation domain.

Read the complete data scientist interview experience from which this question is drawn.