Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 2 of 8

Kernel SVMs and the Kernel Trick

In Lesson 1 the ripe and unripe tomatoes sat in two tidy corners, and a straight line down the middle of the widest gap sorted them. Real data is rarely that polite. Sometimes one class completely surrounds another, and then no straight line can ever separate them, whatever cost C you choose. You will meet exactly that shape in a minute: an espresso bar whose good shots cluster in a tight sweet spot while the bad ones ring the outside, a bullseye no straight line can split.

This lesson shows the idea that rescued the support vector machine: the kernel trick. It lets an SVM bend its boundary into a curve that wraps one class inside another, without ever leaving the two measurements you started with. The interactive below is that same machine from Lesson 1; press Polynomial or RBF and watch the straight line become a closed curve.

By the end of this lesson you will be able to:

  • Explain why some classes cannot be split by any straight line, no matter how large C is
  • Describe the kernel trick: lifting data into a higher-dimensional space where a flat boundary works, using only dot products
  • Tell the polynomial and RBF kernels apart, and tune the two dials C and gamma that trade a tight fit against a smooth one

Prerequisites: Lesson 1 (the maximum margin, support vectors, and the cost C), and you can read a scatter plot.

The problem

Two dials, and a gap no line can cross

Picture an espresso bar. Every shot is set by two dials: the grind (how fine, on a 1 to 30 setting) and the shot time (seconds the pump runs). A shot tastes good only in a narrow sweet spot near grind 15 and time 28. Drift too far in any direction, too coarse or too fine, too fast or too slow, and it tastes bad. So the good shots cluster in the middle, and the bad shots ring the outside.

Each lesson runs in a fresh R session, so we generate those 60 shots right here.

RInteractive R
library(e1071) library(ggplot2) set.seed(1) sweet_grind <- 15 # the ideal grind setting (higher = finer) sweet_time <- 28 # the ideal shot time in seconds # 24 good shots near the sweet spot; 36 bad shots in a ring around it ang <- runif(60, 0, 2 * pi) rad <- c(runif(24, 0, 1.1), 2.2 + runif(36, 0, 1.0)) shots <- data.frame( grind = round(sweet_grind + rad * cos(ang) * 2.4, 1), time = round(sweet_time + rad * sin(ang) * 2.4, 1), taste = factor(c(rep("good", 24), rep("bad", 36))) ) table(shots$taste) #> #> bad good #> 36 24

  

  





Plot the two dials against each other and the shape is unmistakable: a good blob, wrapped by a bad ring.




  

    
RInteractive R

    
ggplot(shots, aes(grind, time, colour = taste)) +
  geom_point(size = 3) +
  labs(title = "Good shots sit in a central blob; bad shots ring the outside",
       x = "grind setting", y = "shot time (seconds)")

    

      
      
    

    

  

  





Now ask Lesson 1's linear SVM to separate them. It draws the best straight line it can, and it is hopeless: a line can put a ring on one side and its centre on the other only by slicing straight through both.




  

    
RInteractive R

    
lin <- svm(taste ~ grind + time, data = shots, kernel = "linear")
mean(predict(lin) != shots$taste)   # fraction of shots misclassified
#> [1] 0.4

    

      
      
    

    

  

  





Forty percent wrong on the very data it trained on. And turning up the cost C will not help: C only makes a straight boundary stricter or softer, it can never make it curve. We need a genuinely different idea.