Anomaly Detection Lesson 5: Time-Series Anomaly Detection

Lesson 5 of 7

Time-Series Anomaly Detection

Every detector so far, from the isolation forest to the autoencoder, judged each point on its own: it looked at a data point's features and asked "does this sit where normal points sit?" Order did not matter; you could shuffle the rows and nothing changed. This lesson breaks that assumption. Here the data arrives in time order, and when a value happens is part of what makes it normal or not.

Follow one running example: Rosa's bakery takes online orders, and Rosa has 16 weeks of daily order counts. Business is growing, weekends are busy, midweek is quiet. Against that steady rhythm, three strange things happen: one Wednesday a post goes viral, one Saturday a snowstorm empties the town, and one week the oven breaks. Each is a different KIND of anomaly, and by the end you will catch all three, in a few lines of base R.

By the end of this lesson you will be able to:

Tell the three kinds of time-series anomaly apart: point, contextual, and collective
Explain why a single flat "mean plus or minus three standard deviations" band is the wrong tool for a series with a trend and a weekly rhythm
Build a robust rolling median / MAD control band and use it to flag point anomalies
Strip out trend and seasonality with STL decomposition and flag the leftover remainder, catching the contextual and collective anomalies the band misses
Score a detector by precision and recall (not accuracy), and tell a one-off anomaly from lasting drift

Prerequisites: you can run R and read base R (indexing, which, sapply, a data.frame); you have done Lesson 1: What Is an Anomaly? (the base-rate trap, why accuracy is the wrong score) and the four point-by-point detectors up to Lesson 4: Autoencoders. Time series, trend, seasonality and the modified z-score are all defined here from scratch.

The idea

Meet Rosa's series

A time series is just a value measured at regular time steps, in order: here, the number of online orders on each of 112 consecutive days. Two patterns run underneath it, and naming them now is what lets us spot the odd days later.

Trend: a slow drift in the overall level. Rosa's bakery is getting more popular, so the baseline climbs gently over the 16 weeks.
Seasonality: a pattern that repeats on a fixed cycle. Rosa's cycle is weekly: quiet Monday to Friday, busy on the weekend. Every seven days the shape repeats.

Each lesson runs in a fresh R session, so let us build Rosa's series right here (run this once). We plant the three anomalies now and study them for the rest of the lesson.

RInteractive R

set.seed(20) day <- 1:112 # 16 weeks of daily online orders dow <- ((day - 1) %% 7) + 1 # day of week: 1 = Mon, ..., 6 = Sat, 7 = Sun season <- c(-8, -12, -9, -5, 3, 28, 23)[dow] # weekly rhythm: quiet midweek, busy weekend trend <- 52 + 0.18 * day # slow, steady growth orders <- round(trend + season + rnorm(112, 0, 4)) # the normal series, plus small noise # now plant three anomalies of three different kinds: orders[45] <- 175 # a Wednesday: a post goes viral (point) orders[76] <- 33 # a Saturday buried by a snowstorm (contextual) orders[92:98] <- orders[92:98] - 30 # the oven broke for a whole week (collective) bakery <- data.frame(day, dow, orders) head(bakery) #> day dow orders #> 1 1 1 49 #> 2 2 2 38 #> 3 3 3 51 #> 4 4 4 42 #> 5 5 5 54 #> 6 6 6 83

Look at the first week: the count creeps up Monday to Friday, then jumps to 83 on Saturday (day 6). That weekend jump is the seasonality. Plot the whole series and you can see both patterns at once: the gentle upward slope (trend) and the repeating weekend spikes (seasonality).

RInteractive R

plot(day, orders, type = "l", col = "grey55", xlab = "day", ylab = "orders", main = "Rosa's daily online orders, 16 weeks") points(day, orders, pch = 16, cex = 0.5, col = "grey30")

One point towers over everything (the viral Wednesday). The other two anomalies are hiding in plain sight, and finding them is the whole job.

Navigate

Time-Series Anomaly Detection

Meet Rosa's series