Design B2C chatbot success metrics and test plan

You are responsible for 'euro-chat', a B2C customer-support chatbot whose goal is to reduce agent handoffs without hurting customer satisfaction. Create a rigorous success framework and testing plan:

1. Propose one conversation-level primary KPI that balances containment (no handoff) against customer utility; give its exact formula and specify the eligible population, attribution window, and how silent/abandoned chats and recontacts within 72 hours should be handled.
2. List at least three guardrail metrics (for example CSAT/NPS, refund/return rate, recontact rate) with thresholds; explain the trade-offs when the primary KPI improves but a guardrail worsens.
3. State the randomization unit (session, user, or intent), the experiment design (A/B, phased rollout), how novelty and learning effects will be handled, and the minimum test duration required given weekday/weekend seasonality.
4. Describe the log instrumentation you need (intents, escalation reason, intent confidence, user authentication status, handoff outcome) and a data-quality plan for detecting labeling drift.
5. If randomization is not feasible, outline a quasi-experiment (for example, difference-in-differences with matched stores) and list the assumptions that must be validated to support causal claims.

Overview: This question tests a data scientist's ability to design product success metrics and a rigorous experiment plan, including KPI formulation, guardrail metrics and thresholds, randomization and experiment design, logging instrumentation, and quasi-experimental causal inference.