Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 8

Heteroskedasticity and Autocorrelation

In Lesson 4, Priya's regression was sabotaged by two columns that said the same thing. This lesson is a subtler kind of trouble: the columns are fine, the slope is fine, and yet the number printed next to it is a lie.

When Priya fits lm(cups ~ temp), R hands her a slope and, right beside it, a standard error: the "give or take" that turns into her p-value and confidence interval. That standard error is computed under two quiet promises about the errors, that they all have the same spread and that they do not lean on each other. Break either promise and the slope stays correct, but the standard error, and every p-value and interval built on it, quietly goes wrong. Usually it comes out too small, so you feel far more certain than you have any right to.

This lesson teaches you to catch both broken promises and to repair the standard error without touching the (perfectly good) slope. The fanning plot below is the first villain: watch the spread grow as you move right.

By the end of this lesson you will be able to:

  • Explain which part of a regression heteroskedasticity and autocorrelation corrupt, and which part they leave alone
  • Test for non-constant variance with the Breusch-Pagan test and fix it with robust (White) standard errors
  • Test for correlated errors with the Durbin-Watson statistic and fix it with Newey-West (HAC) standard errors
  • Choose the right correction for a given dataset

Prerequisites: Lessons 1 to 4 (you can fit a line with lm(), and read its coefficients, standard errors and p-values). Lesson 2 introduced the residual funnel and runs-in-time as shapes; here we make them precise, test them, and correct them. Every new term is defined as it appears.

The number under the estimate

What a standard error is quietly assuming

Priya has expanded her cart and now has 60 trading days of records. A fresh R session starts empty, so we build that table right here.

RInteractive R
set.seed(230) n <- 60 temp <- round(runif(n, 15, 34), 1) # 60 days, each day's high in Celsius noise <- rnorm(n, 0, 1 + 1.15 * (temp - 15)) # the miss GROWS with temperature (see the next step) cups <- round(9 + 1.9 * temp + noise) # sales: a true slope of 1.9 cups per degree, plus noise coffee <- data.frame(temp, cups) head(coffee) #> temp cups #> 1 22.7 49 #> 2 17.2 44 #> 3 25.3 68 #> 4 20.5 51 #> 5 26.9 56 #> 6 17.3 39

  

  





Now fit the line and read the full table, not just the slope:




  

    
RInteractive R

    
fit <- lm(cups ~ temp, data = coffee)
round(summary(fit)$coef, 3)
#>             Estimate Std. Error t value Pr(>|t|)
#> (Intercept)    9.213      6.985   1.319    0.192
#> temp           1.931      0.291   6.624    0.000

    

      
      
    

    

  

  





The slope is 1.931: each extra degree buys about two more cups, exactly the relationship Priya has trusted all course. The column that matters this lesson is Std. Error, the 0.291 next to the slope. It is the estimated give-or-take on that 1.931, and everything downstream, the t value of 6.6 and the tiny p-value, is just the slope divided by its standard error. Formally, for the whole vector of coefficients \(\hat\beta\),



\[ \operatorname{Var}(\hat\beta) = \sigma^2 (X^\top X)^{-1} \]



where \(X\) is the design matrix (a column of 1s and the temp column), \(X^\top X\) is its cross-product, and \(\sigma^2\) is one single number: the variance (the typical squared size) of the error on every day. That one-number-for-all-days assumption is the promise. When Priya's misses are the same size on cool days and hot days, a healthy flat band of residuals like the one below, the formula is right and her standard error of 0.291 is honest.