Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 1 of 8

Support Vector Machines and the Maximum Margin

You have already met classifiers that draw a straight line between two groups, like logistic regression and LDA. A support vector machine draws a straight boundary too, but it chooses that boundary by one strikingly simple and powerful principle: leave the widest possible empty street between the classes.

Below is a support vector machine at work. The circled points, its support vectors, are the only ones that shape the boundary, and switching the kernel bends that boundary to fit. This course is about that machine. This first lesson builds its beating heart on one concrete job a straight line can handle: sorting ripe tomatoes from unripe ones off two cheap sensor readings. Given many lines that all split the two groups, what makes one of them the best?

By the end of this lesson you will be able to:

  • Explain what the margin is and why the widest-margin boundary is the most reliable one
  • Read the boundary equation, the margin, and the max-margin problem, with every symbol defined
  • Find the support vectors, and predict how the boundary moves when a point changes
  • Fit a linear SVM in R, count its support vectors, and tune its cost

Prerequisites: you can fit and read a basic classifier and know what a training set and a decision boundary are; you have seen at least one straight-boundary classifier such as logistic regression or LDA. A scatter plot and a straight-line equation are all the geometry you need.

The setup

Many lines separate them; which do we trust?

A tomato packing line needs to sort fruit into ripe (ship today) and unripe (back to the shelf), automatically, from two cheap sensor readings: redness, a 0 to 10 colour score, and give, how many millimetres the skin dents under a standard press. Ripe tomatoes are redder and softer (high give); unripe ones are greener and firmer (low give). Plotted, they form two clean clouds with an empty strip between them.

Each lesson runs in a fresh R session, so we build a small labelled batch right here: 20 ripe and 20 unripe tomatoes. The three dashed lines below all separate the batch perfectly, every ripe tomato on one side, every unripe one on the other.

RInteractive R
library(ggplot2) set.seed(1) n <- 20 ripe <- data.frame(redness = rnorm(n, 7.2, 0.7), give = rnorm(n, 6.8, 0.7), grade = "ripe") unripe <- data.frame(redness = rnorm(n, 3.4, 0.7), give = rnorm(n, 3.0, 0.7), grade = "unripe") tomatoes <- rbind(ripe, unripe) tomatoes$grade <- factor(tomatoes$grade) ggplot(tomatoes, aes(redness, give, colour = grade)) + geom_abline(slope = -1.00, intercept = 11.2, linetype = "dashed", colour = "grey65") + geom_abline(slope = -0.35, intercept = 7.4, linetype = "dashed", colour = "grey65") + geom_abline(slope = -2.20, intercept = 16.5, linetype = "dashed", colour = "grey65") + geom_point(size = 2.6) + scale_colour_manual(values = c(ripe = "#b5631a", unripe = "#2563a8")) + labs(title = "Three lines that all separate the tomatoes. Which is best?", x = "redness (0 to 10)", y = "give (mm the skin dents)") + theme_minimal(base_size = 13)

  

  





All three lines score 100% on this training batch, yet they are clearly different rules. A tomato that lands in the gap tomorrow could be graded ripe by one line and unripe by another. So "separates the training data" cannot be the whole story. We need a reason to prefer one line over all the others.