Navigate
Courses Tutorials Practice Pricing Tools Upgrade

Handling outliers in a time series

Today we are going to look at one bad day sitting inside an otherwise ordinary time series, and work out exactly what to do about it.

Picture an online store that logs how many orders it gets every day. We have 90 days of that count. On an ordinary weekday the store does close to 240 orders, and on a weekend it settles down to about 150, because fewer people shop then. That weekly up and down is completely normal, and it repeats every 7 days.

Then look at day 61, a Tuesday. The chart below plots all 90 days. Somewhere past the two-thirds mark, the line drops to a single, deep spike down to 38 orders. That is not a mystery either: the store's payment gateway went down for six hours that day, and everyone already knows it.

So the question for this whole lesson is not "how do we find this outlier." We already found it, and we even know why it happened. The real question is what to do with that one point before you fit a model or make a forecast on this data.

Why deleting a point breaks a time series

When you spot a bad value sitting in an ordinary table of unordered rows, say a list of transaction amounts, the easiest fix is often to just drop the row. You lose one data point, and that is the end of it. The rows that are left do not care that a neighbour is gone, because there were no real neighbours to begin with. Order never meant anything.

A time series is different. The order of the rows is not just bookkeeping, it is information. Day 61 is not interchangeable with day 60 or day 62, each one sits on a specific weekday with its own expected level, and the one-day gap between consecutive rows is itself part of what a model learns from.

So what happens if you delete day 61's row outright? Every day after it slides one position earlier. The count that used to belong to day 62, a Wednesday, is now sitting at position 61. But position 61 is still supposed to be a Tuesday, as far as any weekly pattern built from position 1 onward is concerned.

Nothing in the data warns you about this. The series still looks like 89 evenly spaced numbers, so unless you go back and manually realign the calendar, every weekday label from that point on is off by one day.

Let's see this for real on our 90-day series.

RInteractive R
# Build 90 days of orders, then see what happens to the calendar if day 61 is deleted set.seed(410) n <- 90 day <- 1:n wdnames <- c("Fri", "Sat", "Sun", "Mon", "Tue", "Wed", "Thu") weekday_of_day <- function(d) wdnames[((d - 1) %% 7) + 1] weekday <- weekday_of_day(day) is_weekend <- weekday %in% c("Sat", "Sun") base_level <- ifelse(is_weekend, 150, 240) orders <- round(base_level + rnorm(n, mean = 0, sd = 12)) orders[61] <- 38 data.frame(day = 58:64, weekday = weekday[58:64], orders = orders[58:64]) #> day weekday orders #> 1 58 Sat 158 #> 2 59 Sun 161 #> 3 60 Mon 232 #> 4 61 Tue 38 #> 5 62 Wed 245 #> 6 63 Thu 239 #> 7 64 Fri 238 orders_deleted <- orders[-61] true_weekday <- weekday[-61] assumed_weekday <- weekday_of_day(1:length(orders_deleted)) data.frame(position = 58:64, true_weekday = true_weekday[58:64], assumed_weekday = assumed_weekday[58:64]) #> position true_weekday assumed_weekday #> 1 58 Sat Sat #> 2 59 Sun Sun #> 3 60 Mon Mon #> 4 61 Wed Tue #> 5 62 Thu Wed #> 6 63 Fri Thu #> 7 64 Sat Fri

  

  





Look at position 61 onward. true_weekday is the weekday the surviving number actually belongs to. assumed_weekday is what a plain 7-day cycle, restarted from position 1, would call that same position.



The two agree everywhere up to position 60. Starting right at position 61, the row that used to be day 62, they part ways: the real data is a Wednesday's count, but anything downstream that assumes a plain weekly cycle calls it a Tuesday instead.



So deleting day 61 did not just lose one count. It quietly mislabeled every day after it, which throws off the weekly pattern, and any lagged feature (lag 1, lag 7) built from this point on, for the rest of the series.