kNN and the Curse of Dimensionality

You have spent the last courses predicting numbers with regression. Classification is the other half of supervised learning: instead of "how much?", it answers "which kind?". And the most intuitive classifier ever invented needs no equation at all. It just looks at who you sit next to.

Picture a music app that has labeled thousands of past tracks as chill or workout. A brand-new song arrives with no label. To guess its vibe, the app finds the handful of past songs most similar to it and lets them vote. That is k-nearest neighbors, and it is what this lesson builds, then breaks.

By the end you will be able to:

• Explain how kNN labels a new case by the majority vote of its nearest neighbors
• Measure "nearest" with a distance metric, choose k, and run the whole thing in R
• Say why you must scale your features, and why piling on more of them eventually backfires

Prerequisites: you can run R and read its output, and you know what a training set is and what a classifier does (the ML Workflow course).

Known by the company it keeps

Here is the entire idea of kNN in one sentence: to label a new point, find the k training points closest to it and take the majority vote of their labels. No model is fit, no curve is drawn. The training data is the model.

In the chart, each dot is a past song the app already labeled. The red dots (class A) are chill tracks, clustered at low tempo and low energy in the lower left. The blue dots (class B) are workout tracks, up at high tempo and high energy. The square is our new, unlabeled track.

Click anywhere to move the new track around, and slide k. Watch the k closest dots light up, cast their votes, and color the square with whichever class wins. Drop it deep in the red cluster and it is confidently chill; drag it into the blue and it flips to workout; park it on the border and the vote gets interesting.