Regularized Discriminant Analysis
Back in the Classification course's LDA and QDA lesson you met two ways to draw a boundary between Gaussian clouds. LDA assumes every class shares one covariance and draws a straight boundary; QDA gives each class its own covariance and draws a curved one. QDA is more flexible, so it should win, and often it does. But flexibility is paid for in data, and when a lab has four kinds of tea to tell apart from only a dozen samples of each, QDA runs out of data to pay with.
Regularized Discriminant Analysis (RDA) is the dial between them. It starts from QDA's per-class covariance and slides it toward LDA's shared one by however much the data can afford, and often lands somewhere in the middle that beats both ends. The interactive below is that dial in miniature: slide it right for a rigid straight-ish boundary (LDA-like), left for a flexible one that hugs every point (QDA-like), and watch where held-out accuracy is actually highest.
By the end of this lesson you will be able to:
- Explain why QDA overfits when classes are many or the data is thin, and why LDA can be too rigid
- Write RDA's two dials, \(\lambda\) (blend QDA toward LDA) and \(\gamma\) (shrink toward a simple round covariance), as formulas
- Read how the boundary morphs from straight to curved as \(\lambda\) changes, and pick the best \(\lambda\) by cross-validation
Prerequisites: the LDA and QDA lesson (the Gaussian-per-class model, the discriminant \(\delta_k\), and why a shared covariance gives a straight boundary), and the bias-variance trade-off from the ML Workflow course.
When QDA runs out of data
Here is the tension in one number. A covariance matrix for \(p\) features holds \(p(p+1)/2\) free numbers, one variance per feature plus one covariance per pair. LDA estimates one such matrix, pooling every class together. QDA estimates one per class, so for \(K\) classes it must pin down \(K\) times as many.
| Covariances to estimate | For \(K=4\) classes, \(p=2\) features | |
|---|---|---|
| LDA | one shared matrix: \(p(p+1)/2\) | 3 numbers |
| QDA | one per class: \(K \cdot p(p+1)/2\) | 12 numbers |
Three numbers versus twelve. When each class has only a dozen samples, QDA is estimating twelve covariance numbers from thin, noisy data, and a covariance estimated from twelve points is a wobbly thing. Its curved boundary starts bending around accidents of the sample instead of real structure. LDA is steadier because it pools everything into one estimate, but it pays for that steadiness by forcing four genuinely different-shaped clouds to share a single shape.
Let us make that concrete with a running example. A Darjeeling estate grades its tea into four flushes (harvest seasons): first, second, monsoon, and autumn. Grading by taste needs a master taster, so the lab has verified only twelve leaf samples per flush. Each sample carries two cheap machine measurements: briskness (an astringency index) and colour (liquor brightness). Each lesson runs in a fresh R session, so we build that grading log right here.
Plot the two measurements against each other. The four flushes overlap heavily near the centre, and, crucially, they are genuinely different shapes: two long tilted ellipses, one tall narrow one, one round one. That shape difference is what QDA wants to model and LDA is forced to ignore.