Lesson 2 of 3

From one tree to a forest

A single tree overfits and wobbles. The fix is almost embarrassingly simple, and it is one of the most powerful ideas in machine learning.

In Lesson 1 you saw one deep tree hit 96% on training data but only 66% on test, and shift wildly when the data changed. Now you will turn that weakness into a strength.

  • Watch averaging crush a tree's variance, live
  • See why bootstrap samples make trees differ
  • Meet the one trick that makes a forest beat plain bagging

Prerequisites: Lesson 1 (decision trees, and why one deep tree overfits).

The idea

The wisdom of crowds

Ask one noisy expert and you get a noisy answer. Ask hundreds and average them, and the random errors cancel while the real signal survives. That is the whole trick behind a forest.

Key Insight
If each tree has variance \(\sigma^2\) and the trees are independent, the average of \(B\) of them has variance \(\sigma^2 / B\). More trees, steadier predictions: the random errors average away.

The catch is hiding in one word: independent. Hold that thought.