Derive and compare core ML and RL methods

Work through the following machine learning fundamentals rigorously: state the assumptions, give the equations, and justify the trade-offs.

1. Derive the update rules for full-batch gradient descent (GD) and stochastic gradient descent (SGD) for $L(w)=\frac{1}{n}\sum_{i=1}^{n}\ell_i(w)$. Compare their convergence behavior, gradient variance, and wall-clock efficiency, and explain the conditions under which SGD outperforms GD.

2. Define batch size. Given $n=50{,}000$ samples, 5 epochs, and batch size $b=200$, compute the number of update steps per epoch and in total. If $b$ is increased to 2,000, compute the new step counts and propose a learning-rate adjustment based on linear scaling; explain when this rule fails.

3. Classify each of the following algorithms as supervised or unsupervised and give one use case for each: logistic regression, SVM, k-NN, k-means, PCA, t-SNE, and Isolation Forest.

4. Explain how reinforcement learning relates to supervised and unsupervised learning. Write the REINFORCE gradient $\nabla_\theta J = \mathbb{E}\left[\sum_t \nabla_\theta \log \pi_\theta(a_t|s_t) G_t\right]$ and show how a baseline $b_t$ keeps the estimator unbiased while lowering variance; express the gradient for a length-3 trajectory with returns $G=[3,1,-1]$ and score-function terms $g_1,g_2,g_3$ using a constant baseline $b=\text{mean}(G)$.

5. Explain how neural networks are used in RL, for example in DQN, policy gradient, and actor-critic methods. For DQN, describe why target networks and experience replay make training more stable, and give a failure mode that occurs without them.

6. Compare Transformers and RNNs in terms of parallelism, long-range dependency handling, and complexity. For sequence length $n=1024$ and model dimension $d=512$, estimate the asymptotic time and memory cost of self-attention, and name two techniques that reduce the quadratic scaling.

7. Define embeddings and polysemy. Propose a method to distinguish “King” in chess contexts from monarchy contexts using contextual encoders or multi-sense embeddings; outline an intrinsic evaluation based on word sense disambiguation (WSD) and an extrinsic evaluation based on downstream accuracy.

8. Given a single 24-GB GPU and a 7B-parameter model, design a low-compute fine-tuning plan, such as QLoRA or adapters, 4-bit quantization, gradient checkpointing, and mixed precision. Choose a LoRA rank $r$ and specify batch size, sequence length, optimizer, and learning-rate schedule. Provide a back-of-envelope estimate of trainable parameters assuming hidden size $\approx 4096$ and about 32 layers; state any assumptions about which projections you adapt.

Overview: This question tests a candidate's command of core machine learning and reinforcement learning ideas, including optimization (gradient methods and batch-size trade-offs), supervised versus unsupervised algorithms, policy-gradient RL and variance reduction, deep RL stabilization techniques, sequence model complexity (Transformers versus RNNs), embeddings and polysemy, and low-compute fine-tuning strategies. It is often used to assess both theoretical understanding and practical engineering judgment around convergence, variance, computational and memory complexity, representation learning, and resource-constrained model adaptation; the domain is Machine Learning, and the assessment covers conceptual understanding and practical application.

Read the full data scientist interview experience this question came from.