Estimate b when features exceed samples

Take the linear model $y = Xb + \varepsilon$, where $X \in \mathbb{R}^{n \times (m+1)}$ and an intercept column is included.

a) Work out the OLS estimator $\hat{b} = (X^TX)^{-1}X^Ty$, and state both the rank conditions required for identifiability and the sampling distribution of $\hat{b}$ under the classical assumptions.

b) Suppose instead that $m > n$. Lay out at least three workable strategies (for instance ridge: $\hat{b}_{ridge} = (X^TX + \lambda I)^{-1}X^Ty$; lasso; elastic net; forward selection; PCA/PLS), and for each one explain how you would pick $\lambda$ and how you would verify generalization (including the specifics of cross-validation).

c) Under what circumstances does the Moore–Penrose pseudoinverse produce a sensible minimum-norm solution, and what drawbacks come with it?

d) Explain why naively duplicating rows does not fix rank deficiency and can also damage inference.

Overview: This question gauges command of linear regression theory — identifiability and the sampling distribution of OLS — alongside high-dimensional competencies such as regularization, variable selection, dimensionality reduction, the properties of the Moore–Penrose pseudoinverse, and the statistical consequences of naive upsampling.

Read the full Data Scientist interview experience this question came from