Explain Ensemble Learning and Its Principal Families

Describe why aggregating several models can improve predictions. Contrast bagging, random forests, boosting, voting/averaging, and stacking. For each family, cover how its members are trained, how their predictions are merged, which type of error it mainly reduces, and the key leakage or overfitting risks.

Constraints & Assumptions

• Work under supervised learning with a fixed train/validation/test split.
• Depending on the method, base learners may be homogeneous or heterogeneous.
• The comparison should cover both regression and classification where relevant.
• Any meta-learner must be trained without using in-sample base predictions as if they were out-of-sample.

Clarifying Questions to Ask

• Is the dominant issue variance, bias, calibration, or robustness?
• Can base learners be trained independently, or is sequential training acceptable?
• Are inference latency and model size constrained?

Hint — Look at correlated errors: Averaging provides little benefit when many identical models make the same mistake.

Hint — Protect the stack: Build meta-features from folds in which each base prediction is produced by a model that did not see that row during training.

What a Strong Answer Covers

• Diversity and correlated errors as the reason aggregation can help.
• Clear distinctions among parallel bagging, randomized forests, sequential boosting, and stacking.
• Out-of-fold training for a stacking meta-model.
• Trade-offs across accuracy, interpretability, calibration, latency, and correlated failures.

Follow-up Questions

• Why can boosting overfit noisy labels even while training loss continues to improve?
• How would you distill a costly ensemble into a single faster model?

Overview: Explain why combining several models can lead to better predictions. Address data and labels, leakage-safe features, baselines and model choice, offline evaluation, deployment constraints, monitoring, and drift.