Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 3 of 6

Discriminant Analysis, LDA and QDA

Lesson 1 classified by distance to neighbors. Lesson 2 reasoned with probabilities but pretended every feature was an independent clue. Discriminant analysis keeps the probability idea and drops that pretence: it lets features move together, models each class as a tilted cloud, and then draws an explicit line between the clouds.

Picture a cannery's sorting belt. A camera measures every passing fish twice: its length in centimeters, and its lightness on a 0 (dark) to 10 (pale) scale. Salmon run long and pale; sea bass run short and dark. Plotted, the two species form two overlapping clouds, and the machine needs one rule to split the belt: salmon to the left, sea bass to the right. This lesson draws that boundary, two different ways.

By the end of this lesson you will be able to:

  • Model each class as a Gaussian cloud and label a new fish by which cloud most likely produced it
  • Explain why LDA's shared-covariance assumption draws a straight boundary, and why QDA's per-class covariance lets it curve
  • Choose between LDA and QDA, and run both in R

Prerequisites: you can run R and read its output, you know what a training set and a classifier are (the ML Workflow course), and Lessons 1 (kNN) and 2 (Naive Bayes). Mean, variance and correlation will be enough; the covariance matrix is defined as it appears.

The picture above is one boundary splitting the two clouds, with a dial that makes it more or less flexible. Discriminant analysis draws two specific kinds of boundary, a straight one and a curved one, and the rest of this lesson is about where each comes from and which to trust.

The idea

Model each class as a cloud

Naive Bayes already taught the move: to label a point, ask which class makes the evidence most likely, weighted by how common each class is. Discriminant analysis uses the very same rule, with one upgrade. Instead of treating length and lightness as separate, independent clues, it describes each species with a single 2D bell-shaped cloud that has a center and a spread, and crucially can be tilted, so that longer fish also tend to be paler.

Each lesson runs in a fresh R session, so we build a labeled catch right here: 120 salmon and 120 sea bass, each a point in the length-lightness plane. The two clouds are deliberately different shapes, which will matter in a moment.

RInteractive R
library(MASS) set.seed(1) n <- 120 # Salmon: a compact cloud, longer and paler; length and lightness rise together. salmon <- mvrnorm(n, mu = c(70, 6.0), Sigma = matrix(c(25, 3.0, 3.0, 0.8), 2)) # Sea bass: shorter, darker, with a much wider and differently tilted spread. bass <- mvrnorm(n, mu = c(58, 4.2), Sigma = matrix(c(80, -2.0, -2.0, 0.5), 2)) fish <- data.frame( species = factor(rep(c("salmon", "bass"), each = n)), length = c(salmon[, 1], bass[, 1]), # centimeters lightness = c(salmon[, 2], bass[, 2]) # 0 = dark, 10 = pale ) table(fish$species) #> #> bass salmon #> 120 120

  

  






  

    
RInteractive R

    
library(ggplot2)
ggplot(fish, aes(length, lightness, color = species)) +
  geom_point(alpha = 0.7, size = 2) +
  scale_color_manual(values = c(bass = "#2563a8", salmon = "#b5631a")) +
  labs(title = "Two species, two clouds",
       x = "length (cm)", y = "lightness (0 = dark, 10 = pale)") +
  theme_minimal(base_size = 13)

    

      
      
    

    

  

  





Each cloud is a multivariate Gaussian: the 2D version of the familiar bell curve. It is described completely by two things. The mean vector \(\mu_k\) is the cloud's center, the average length and lightness of class \(k\). The covariance matrix \(\Sigma_k\) describes its shape: how wide it spreads along each feature, and whether the two features lean together (a tilt). Its full formula, for a point \(x\) with \(p\) features, is



\[ f_k(x) = \frac{1}{(2\pi)^{p/2}\,|\Sigma_k|^{1/2}}\,\exp\!\left(-\tfrac{1}{2}(x-\mu_k)^\top \Sigma_k^{-1}(x-\mu_k)\right) \]



where \(x\) is the feature vector (here length and lightness, so \(p = 2\)), \(\mu_k\) is the class center, \(\Sigma_k\) the class covariance, \(|\Sigma_k|\) its determinant, \(\Sigma_k^{-1}\) its inverse, and \(\top\) means transpose. You will never compute this by hand. All it says is: a point scores high under class \(k\) when it sits near that class's center, measured in a way that accounts for the cloud's spread and tilt.