Prove and apply statistical ML fundamentals

Solve these statistical machine-learning exercises using precise mathematics and small numerical computations.

1. Starting from first principles, derive ordinary least squares for linear regression: the model, its assumptions, the normal equations, the closed-form estimator, the conditions needed for $(X^T X)^{-1}$ to exist, and the ridge solution; then explain the bias-variance effects.

2. For logistic regression, write the negative log-likelihood for binary labels, derive the gradient and Hessian, and prove convexity. Then perform one explicit gradient step with no bias term, learning rate 0.5, input $x=(1,2)$, label $y=1$, and current weights $w=(0.1,-0.2)$.

3. For overfitting, list three distinct mitigation techniques—regularization, early stopping, and data augmentation are possible examples—and explain when each helps or hurts; propose a cross-validation plan for tuning $\lambda$ in $L_2$ regularization.

4. Bootstrapping versus boosting: a) Bootstrapping: given sample values $[2,3,5,7,11]$, describe the percentile-interval procedure for the mean; show the first two bootstrap resamples you would draw (with replacement) and compute their means; explain why the bootstrap can estimate uncertainty without parametric assumptions. b) Boosting: explain the core idea of sequentially fitting to residuals or reweighted errors. Perform one AdaBoost step with three training points that start with equal weights of $1/3$ each, where the weak learner misclassifies only the second point: compute $\epsilon$, $\alpha = \frac{1}{2}\ln\left(\frac{1-\epsilon}{\epsilon}\right)$, the unnormalized updated weights, and the normalized distribution for the following round.

5. Compare bagging, boosting, and random forests with respect to bias, variance, and robustness to noisy labels; provide one scenario where each is preferable.

Overview: This question assesses mastery of statistical machine-learning fundamentals—linear and logistic regression derivations, regularization and the bias–variance trade-off, bootstrap resampling, boosting algorithms, and ensemble comparisons—using precise mathematical reasoning and small numerical computations.

Read the full Data Scientist interview experience this question came from.