Detect leakage and evaluate a prediction model

You are handed a churn prediction model built to estimate whether a user will place an order within the next 28 days. The training set contains weekly-computed features, and some of them expose post-label information—for instance, features that count activity inside the prediction window.

(a) List at least five concrete leakage sources in a logs-based feature set (such as features built from orders inside the label window, coupon_applied over the next 28 days, post-treatment delivery ETA, or support contacts influenced by the label). For each one, rewrite the feature so it can be computed at prediction time using a strict event-time cutoff.

(b) Suggest a time-based cross-validation design (rolling origin) and specify train/validation/test splits that prevent future leakage. Explain how you would treat users who enter or leave the cohort and cold-start users.

(c) Offline evaluation gives $AUC=0.79$ and $PR\text{-}AUC=0.23$. Online, targeting the top decile lifts conversion by $1.5\%$ but increases cancellations by $0.3$ percentage points. Define business KPIs and a cost-sensitive objective for tuning the decision threshold; include calibration (Platt/Isotonic) and describe how you would monitor calibration drift in production.

(d) Interpretation: a standardized feature past_7d_orders has a logistic regression coefficient of $0.40$ and the baseline log-odds of churn is $-1.50$. Compute the odds ratio for a +1 SD increase and the resulting change in churn probability from the baseline. Discuss the limits of such ceteris paribus interpretations when features are correlated.

(e) Outline a monitoring plan covering data quality, feature distributions, PSI, label delay, and prediction drift, along with a retraining policy triggered by performance and covariate shift.

Overview: This question tests the ability to build production-grade supervised machine learning systems—finding and fixing data leakage in log-based features, creating time-aware validation, setting KPI-driven thresholds and calibration, interpreting model coefficients, and defining monitoring and retraining policies for a churn model in the Machine Learning domain. It is frequently used because interviewers need to evaluate both conceptual knowledge of leakage, validation, and calibration and practical skill in deploying and maintaining models with cost-sensitive objectives and production monitoring, so the required level spans conceptual reasoning and hands-on implementation.