Random Forest
A random forest is Bagging applied to Decision Trees with extra feature-subsampling at each split.
Why subsample features if bagging already randomizes data?
Pure bagging can leave one very informative feature dominating every tree, so the bootstrapped trees stay highly correlated. Random feature subsets at each split decorrelate them further.
- Standard decision tree: at each split, look at all features and pick the best
- Random forest: look at a random subsample of features and pick the best among those
- Typical choice:
- Resample the feature subset independently for each split
Intuition
Bagging’s variance reduction only works if the trees are uncorrelated. If one feature is way more predictive than the rest, every bootstrap tree splits on it first and they all end up near-clones. Hiding that feature at most splits (by random subsetting) forces different trees to find different paths through the data, which is what makes the average actually smooth things out. You deliberately make each tree slightly worse so the ensemble becomes much better.