Lesson 2 of 3
From one tree to a forest
A single tree overfits and wobbles. The fix is almost embarrassingly simple, and it is one of the most powerful ideas in machine learning.
In Lesson 1 you saw one deep tree hit 96% on training data but only 66% on test, and shift wildly when the data changed. Now you will turn that weakness into a strength.
- Watch averaging crush a tree's variance, live
- See why bootstrap samples make trees differ
- Meet the one trick that makes a forest beat plain bagging
Prerequisites: Lesson 1 (decision trees, and why one deep tree overfits).
The idea
The wisdom of crowds
Ask one noisy expert and you get a noisy answer. Ask hundreds and average them, and the random errors cancel while the real signal survives. That is the whole trick behind a forest.
Key Insight
If each tree has variance \(\sigma^2\) and the trees are independent, the average of \(B\) of them has variance \(\sigma^2 / B\). More trees, steadier predictions: the random errors average away.
The catch is hiding in one word: independent. Hold that thought.