Select interest thresholds under skewness and cost

Suppose each user has an interest_score in $[0,1]$ for a new feature, and the observed distribution looks right-skewed. You can reach no more than K users each week. Describe a principled way to choose a cutoff (for example, the 75th or 90th percentile) that maximizes incremental value given a per-contact cost $c$ and an expected benefit function $b(\text{score})$. Explain how you would (a) estimate $b(\text{score})$ from historical data, such as with isotonic or spline calibration; (b) compute the profit-maximizing cutoff using $\text{expected lift} \times \text{margin} - c$; (c) adapt if the distribution is actually left-skewed; (d) reduce instability from sampling noise, for example via Bayesian shrinkage or percentile confidence intervals; and (e) set up an ongoing backtest to validate the threshold against alternatives. Provide formulas and a step-by-step selection algorithm.

Overview: The question tests a data scientist's ability to perform profit-oriented threshold selection with calibrated interest scores, uplift estimation, and decision-theoretic optimization under cost and weekly capacity limits.

Read the full data scientist interview experience this question came from.