Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 7 of 13

Choosing Smoothness in a GAM

In Lesson 6, Priya's straight line was blind to the temperature hill, and a GAM using the same predictor explained 54% of her ice-cream sales instead of 0.6%. You wrapped a predictor in s(), and the wiggliness penalty quietly chose how much to bend. It felt like magic: no polynomial degree to guess, no knots to place.

But a GAM is not tuning-free, and trusting it blindly is how you ship a curve that is secretly wrong. There was a number you accepted without looking: k, the basis size. Was it big enough? And when Priya adds a second predictor, how does she know the two smooths are telling her separate things instead of fighting over the same signal?

This lesson is the diagnostic half of GAMs. You will learn to interrogate a smooth the way you already interrogate a linear model's residuals: is it flexible enough, are its assumptions holding, and are its predictors truly separate.

By the end of this lesson you will be able to:

  • Tell k (the flexibility ceiling you set) apart from edf (the flexibility the penalty actually spends), and spot when the ceiling is too low
  • Read gam.check(): its basis-size table and its four residual plots
  • Raise k safely, knowing the penalty still protects you from overfitting
  • Recognize and measure concurvity, the smooth-curve version of collinearity, and act on it

Prerequisites: you can fit a GAM and read its edf (GAMs, Splines and Smooths, Lesson 6), you know the bias-variance tradeoff (underfit vs overfit), and you can read a linear model's residual plots. It helps to have met multicollinearity: two predictors carrying the same information.

The dial below is the one from Lesson 6. Slide it left and the smooth is too stiff; slide it right and it chases noise. That slider IS the number k. This lesson is about finding the right setting on purpose, and proving you found it.

The knob you did not look at

The dial has a ceiling

This month Priya kept a finer log. For 400 visits she noted the hour of day and how many ice creams she sold in that hour, plus the temperature at that moment. Let us build her log inline (each lesson runs in a fresh R session, so all the data lives right here) and look at the first few rows.

RInteractive R
library(mgcv) set.seed(1) n <- 400 cart <- data.frame(hour = round(runif(n, 6, 22), 1)) # hour of day, 6am to 10pm peak <- 30 * exp(-(cart$hour - 9)^2 / 1.5) + # a morning-commute rush 40 * exp(-(cart$hour - 18)^2 / 2.5) # a bigger evening rush cart$sales <- round(pmax(0, 45 + peak + rnorm(n, 0, 7))) # ice creams sold that hour cart$temp <- round(15 + 1.4 * (cart$hour - 6) - # temperature climbs through 0.05 * (cart$hour - 6)^2 + rnorm(n, 0, 0.8), 1) # the day, then eases head(cart, 4) #> hour sales temp #> 1 10.2 59 19.7 #> 2 12.0 57 22.8 #> 3 15.2 58 24.1 #> 4 20.5 46 25.2

  

  





Her sales across the day are not one hill but two: a morning-commute bump around 9am and a bigger evening rush around 6pm, with a lull in between. That is a genuinely wiggly shape. Fit the obvious GAM, a smooth of hour, and read the smooth term.




  

    
RInteractive R

    
g <- gam(sales ~ s(hour), data = cart, method = "REML")
round(summary(g)$s.table, 2)
#>          edf Ref.df     F p-value
#> s(hour) 8.81   8.99 123.9       0

    

      
      
    

    

  

  





That looks wonderful. The smooth is wildly significant, and (not shown) it explains 74% of the variation in sales. You might stop here. But look closely at the two numbers and ask where they came from. The edf is 8.81: the fitted curve spent about 8.8 straight-lines'-worth of flexibility. The Ref.df is 8.99. Those two are suspiciously close, and 8.99 is not an accident.



When you write s(hour), mgcv uses a default basis of k = 10 building-block curves. That k is a ceiling: it caps how wiggly the smooth is allowed to get. The penalty then chooses the actual wiggliness up to that ceiling. In symbols, the effective degrees of freedom are boxed in:



\[ 1 \le \text{edf} \le k - 1 \]



The lower bound, edf near 1, is a straight line. The upper bound, \(k-1\), is the wiggliest the basis can express (mgcv reserves one degree for the overall level, so the usable ceiling is \(k-1 = 9\), which it reports as k'). Our edf came back at 8.81, pressed right up against that ceiling of 9. That is the tell we need to chase down next: is 8.81 the honest answer, or just as high as the ceiling would let the curve climb?