Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 4 of 7

Hyperparameter Tuning Strategies

In Lesson 3 you quietly ran a search: eight polynomial degrees, scored one by one, keep the best. That is a grid search, the simplest tuning strategy there is. It works fine for one knob and eight settings. But the bakery has grown. You now predict daily cake sales from six signals, not one, and you are fitting a tree with two knobs worth turning. Trying every combination is no longer cheap, and soon it is impossible. This lesson is about searching well: how to find good settings without burning your whole budget on bad ones.

By the end you will be able to:

  • Tell a hyperparameter (a knob you set before training) from a parameter (a number the model learns)
  • Explain why, for the same number of tries, random search matches or beats a grid, and when Bayesian search beats both
  • Set up a search in R and spend a limited tuning budget where it actually pays off

Prerequisites: Lessons 1 to 3 (k-fold cross-validation, and that "tuning" means trying several settings and keeping the best). You can fit a model and call predict().

Two kinds of number

Parameters the model learns, hyperparameters you choose

Picture baking a cake. You set the oven dial to 180 degrees and a timer to 40 minutes: those are choices you make before anything bakes. What comes out, how risen, how browned, how moist, is decided during baking by the batter reacting to that heat. You do not set the browning directly; you set the dial, and the browning follows.

A model works the same way, and the two words for it matter for the rest of this lesson:

  • A parameter is a number the model learns from the data while it trains. You never set these by hand. In a decision tree they are the actual split questions inside the tree ("is promo on?", "is temperature above 24?"), chosen automatically to fit the bakery's history.
  • A hyperparameter is a number you set before training, that shapes how the model learns. It is the oven dial. The model cannot learn it from the data, because it governs the learning itself.

Let us build the bakery's data and meet its hyperparameters. Each lesson runs in a fresh R session, so we create everything inline (run this once):

RInteractive R
# 300 days of the bakery: daily cake sales vs six signals (simulated so we control the truth). set.seed(1) n <- 300 bakery <- data.frame( temp = round(runif(n, 5, 35), 1), # the day's high, degrees C weekend = rbinom(n, 1, 2/7), # 1 if Saturday or Sunday promo = rbinom(n, 1, 0.30), # 1 if a promotion ran that day holiday = rbinom(n, 1, 0.08), # 1 if a public holiday rain_mm = round(rexp(n, 1/4), 1), # rainfall, mm passers = round(runif(n, 50, 400)) # foot-traffic index outside the shop ) bakery$sales <- with(bakery, 40 + 6*temp - 0.15*temp^2 + 25*weekend + 35*promo + 60*holiday - 1.3*rain_mm + 0.04*passers + rnorm(n, 0, 12)) # a curve + effects + noise (sd 12 cakes) dim(bakery) #> [1] 300 7

  

  





Now grow one small tree and read what it did. We chose maxdepth = 2 (a hyperparameter, at most two questions deep); the tree then learned everything else, which signals to split on and where:




  

    
RInteractive R

    
library(rpart)
# maxdepth = 2 is a HYPERPARAMETER we set. The splits below are PARAMETERS the
# tree LEARNED from the data: which signal to ask about, and at what cutoff.
# (cp = 0 just turns off rpart's built-in pruning, so maxdepth is the only size
# limit here; it is plumbing, not a third knob to tune.)
small <- rpart(sales ~ ., data = bakery,
               control = rpart.control(maxdepth = 2, cp = 0))
small
#> n= 300
#>
#> node), split, n, deviance, yval
#>       * denotes terminal node
#>
#> 1) root 300 291637.90 116.9736
#>   2) promo< 0.5 204 163850.20 106.3624
#>     4) holiday< 0.5 183  85148.30 100.1916 *
#>     5) holiday>=0.5 21  11008.73 160.1365 *
#>   3) promo>=0.5 96  56006.90 139.5223
#>     6) weekend< 0.5 67  26504.99 131.2315 *
#>     7) weekend>=0.5 29  14256.40 158.6770 *

    

      
      
    

    

  

  





You set one number, maxdepth = 2. The tree discovered the rest on its own: split first on promo, then on holiday or weekend, and predict an average for each of the four leaves. Those splits and averages are the parameters, learned. Grow it deeper and it would also learn temperature cutoffs like "is temperature above 24?". What it can never learn is the dial itself: how deep to grow, how large a leaf to allow. Those you must choose, and choosing them well is the whole game of tuning.



Think of the knobs as axes. Every setting is a point on the floor; the height above each point is that setting's cross-validated error. That landscape is the search space, and formally we want its lowest point:



\[ \lambda^{*} = \arg\min_{\lambda \,\in\, \Lambda}\ \mathrm{CV}(\lambda) \]



Here \(\lambda\) (lambda) is one combination of hyperparameters (say, maxdepth = 4, minbucket = 10), \(\Lambda\) is the whole set of combinations you would consider, \(\mathrm{CV}(\lambda)\) is that combination's cross-validated error (the honest score from Lessons 1 to 3), and \(\lambda^{}\) is the winner. The catch is that every time you evaluate \(\mathrm{CV}(\lambda)\) you must train and score a whole model, which costs time. So you get a budget \(B\): a fixed number of settings you can afford to try. Tuning strategy is simply how you spend* \(B\).



Key Insight
Parameters are learned; hyperparameters are chosen. Tuning is the search for the best choice, and because each try costs a full model fit, the real question is never "which setting is best?" but "how do I find a great setting within a limited number of tries?"