Outlier Detection Methods

In Lesson 3 you caught Maya's $905 street-festival day with a boxplot and the 1.5 x IQR rule, then chose, on purpose, to keep it with robust summaries. That worked because you could SEE the lone dot. But real data does not always announce itself: some outliers hide behind a second extreme value, and some are invisible to any single-column rule, surfacing only when you look at two variables at once.

This lesson hands you the numeric tools the eye cannot supply. The scatter below plots all 30 of Maya's March days, daily revenue against the number of customers through the door (foot traffic). Most days climb a tidy line, more customers, more revenue. A few do not, and by the end you will have a rule for each kind.

By the end of this lesson you will be able to:

• Flag outliers with the z-score, and explain the trap that makes it miss real ones
• Use the robust modified z-score (built on the median and MAD) to catch what the z-score masks
• Spot a multivariate outlier, a day that is normal on every variable alone but wrong in combination, with Mahalanobis distance
• Decide what to do, keep, cap or drop, and measure exactly how each choice moves your numbers

Prerequisites: you can run R, and you have met one-variable EDA (the histogram, mean vs median, the IQR and the boxplot) in Lesson 1, correlation in Lesson 2, and the keep/cap/drop decision in Lesson 3. Every new term is defined as it appears.

The z-score: how many standard deviations out?

The boxplot is a picture; to flag outliers across a table with fifty columns you need a number. The oldest one is the z-score. It rephrases "how far is this value from the rest?" as "how many standard deviations from the mean does it sit?"

Write \(x\) for a single day's revenue, \(\bar{x}\) (read "x-bar") for the mean revenue across all the days, and \(s\) for the standard deviation, the typical distance of a day from that mean. The z-score of that day is

\[ z = \frac{x - \bar{x}}{s} \]

A common rule of thumb flags any day with \(|z| > 3\): more than three typical distances from the centre. Each lesson runs in a fresh R session, so let us build Maya's March here (run this once), then read off the centre and spread:

Now apply the rule. We look at the last three days in particular: day 28 is the $905 festival, day 29 a $470 catering order, day 30 a $395 wedding pre-order.

The festival day screams out at \(z = 4.71\). But look again: only one day is flagged. The genuinely busy $470 catering day, which the boxplot's upper fence of $405 clearly marks as an outlier, slips through at \(z = 1.35\), nowhere near 3. Why?

Because the z-score measures each day against a mean and standard deviation that the outliers themselves have already inflated. That single $905 day dragged \(s\) up to about 130. Drop it and recompute the catering day's z-score:

Now it clears the bar at \(z = 3.27\). The festival day was masking the catering day: one extreme value swelled the standard deviation so much that a second real outlier looked ordinary. This is the z-score's fatal flaw, and it bites hardest exactly when you have more than one outlier, which is most of the time. The histogram makes the lopsidedness plain, the festival stranded alone in the far-right bin.