Explain decision trees and tree ensembles

Prompt

1. Describe the way a decision tree carries out classification or regression.
2. What criterion does the tree use to pick a split, and what objective functions apply to classification versus regression?
3. Identify the main hyperparameters and explain how they influence the bias–variance trade-off.
4. Choose another machine learning algorithm that is built on decision trees—for example, Random Forest or Gradient Boosted Trees—and explain how it works and the scenarios where you would select it over a single tree.

Overview: This question tests understanding of how decision trees work, the split criteria used for classification and regression, the way hyperparameters affect the bias–variance trade-off, and the mechanics and rationale behind tree-based ensemble methods in machine learning.