Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 6 of 8

Bayesian Optimization for Hyperparameters

In Lesson 5 you blended several strong models into a stack, and every one of them ran on settings we took on faith. This lesson is about choosing those settings when checking a single candidate is painfully expensive.

Meet Mara, a data scientist at an insurance company. Her fraud-detection model is an RBF SVM (Lessons 1 and 2), and she needs to tune its kernel width, gamma. On her real dataset of 40,000 claims, scoring ONE candidate gamma means one full 5-fold cross-validation: about 25 minutes. A modest 20-value grid over gamma alone is more than 8 hours; add a grid over cost and it is a week of compute. Her budget before Friday's model review: roughly 8 evaluations, overnight. Eight questions to an oracle that takes 25 minutes per answer. Which eight should she ask?

Bayesian optimization (BO) answers by spending the budget like a scientist: build a cheap model OF the search itself, use it to predict where a better setting probably hides, evaluate exactly there, and learn from every result. By the end of this lesson you will be able to:

  • Frame hyperparameter tuning as expensive black-box optimization, and say precisely why grid and random search waste a small budget
  • Fit a GP surrogate to a handful of (setting, CV score) pairs and read its mean and band as everything the search knows so far
  • Compute Expected Improvement, the rule that decides where the next evaluation goes, and explain how it trades exploitation against exploration
  • Run the full BO loop in R, read its evaluation history, and judge when BO earns its keep over blind search

Prerequisites: the RBF SVM and its gamma dial (Lesson 2), the GP posterior mean and its honest band (Lesson 4, the engine of this lesson), and k-fold cross-validation as an accuracy estimate.

The interactive below is the whole lesson in one loop. The dashed curve is the true objective, which in real life you can never afford to trace. The three dots are evaluations already paid for; the blue curve and band are the surrogate; the lower panel is the acquisition score that picks the next evaluation. Press Next sample a few times and watch it home in on the global peak, then keep pressing and watch it double-check the runner-up bump.

The problem

One candidate costs one full CV run

First, put the problem in your hands. Gamma, from Lesson 2, sets how far one training claim's influence reaches: a small gamma makes broad, smooth boundaries; a large gamma wraps tight islands around individual points. Somewhere between underfitting and overfitting sits the gamma Mara wants, and the only way to score a candidate is to run the full cross-validation. That makes tuning a black-box optimization problem: there is no formula for the curve Mara is climbing and no gradient to follow, only the right to ask, at full price, "what is the score at this setting?"

Each lesson runs in a fresh interactive R session, so we build a small replica of Mara's problem right here: 300 claims, two features, and a known fraud pattern (late-night filings of mid-size claims). Small enough that one evaluation takes about a second in your browser instead of 25 minutes, so you can afford to play. The mechanics are identical.

RInteractive R
library(e1071) library(ggplot2) set.seed(42) n <- 300 claims <- data.frame( hour = round(runif(n, 0, 24), 1), # hour of day the claim was filed amount = round(runif(n, 0.1, 10), 2) # claim amount, thousands of dollars ) # fraud concentrates in a pocket: filings around 2 am, claims around 6 thousand d_pocket <- sqrt((claims$hour - 2)^2 / 4.5^2 + (claims$amount - 6)^2 / 3^2) claims$fraud <- factor(ifelse(runif(n) < plogis(5 - 4 * d_pocket), "yes", "no")) table(claims$fraud) #> #> no yes #> 231 69

  

  





Now the expensive function itself. It takes one candidate setting and returns one number: the 5-fold cross-validated accuracy of the SVM at that gamma. We search on the log scale, because gamma ranges over five orders of magnitude (0.001 to 100) and "nearby settings" only means anything on the log axis. Cost stays fixed at 1 so the picture stays one-dimensional; everything below works unchanged with more dials.




  

    
RInteractive R

    
cv_acc <- function(log_gamma) {
  folds <- rep(1:5, length.out = nrow(claims))
  hits <- 0
  for (k in 1:5) {
    fit <- svm(fraud ~ hour + amount, data = claims[folds != k, ],
               kernel = "radial", gamma = 10^log_gamma, cost = 1)
    hits <- hits + sum(predict(fit, claims[folds == k, ]) == claims$fraud[folds == k])
  }
  hits / nrow(claims)
}

X <- c(-2.5, 0.5, 1.5)          # log10(gamma) of three spread-out scouting runs
Y <- sapply(X, cv_acc)
data.frame(log_gamma = X, gamma = 10^X, cv_accuracy = round(Y, 4))
#>   log_gamma        gamma cv_accuracy
#> 1      -2.5  0.003162278      0.7700
#> 2       0.5  3.162277660      0.8900
#> 3       1.5 31.622776602      0.8467

    

      
      
    

    

  

  





Three evaluations spent, five left. The tiny gamma scores 0.77, which is exactly the share of non-fraud claims: that SVM predicts "no" for everyone. The middle candidate is the best so far at 0.89. And here is the question that defines this lesson: where do evaluations 4 through 8 go?



The standard answers are grid search (predetermined lattice) and random search (draws from the range). The interactive below shows both spending a 16-evaluation budget on a two-dial version of this problem. Notice the one thing they have in common: candidate number 9 is chosen exactly as if results 1 through 8 had never happened.




Key Insight
Grid and random search are blind: they decide every candidate before seeing a single result. Mara has already paid for three answers, and those answers contain real information, such as "the far left is hopeless". A search that cannot use what it has already learned is wasting the most expensive thing she has.