Gaussian Processes for Regression

In Lesson 3 you steadied a classifier on thin data by blending two covariance estimates. This lesson stays in the thin-data world but changes jobs: regression, where the model must not only predict a number but admit how sure it is.

Meet Asha, a process engineer at a small ceramics factory. She needs the kiln temperature that maximizes glaze hardness. Each test firing ties up the kiln for a full day, so her entire dataset is six firings: temperatures from 880 to 1180 degrees Celsius, hardness measured in Vickers (HV) for each. Between 1000 and 1140 degrees there is a 140 degree stretch where she has never fired at all. Before she books another kiln day, she wants two things from a model: its best guess at every untried temperature, and an honest admission of where that guess is shaky.

That is exactly what a Gaussian process (GP) delivers: a prediction curve plus a band that pinches tight at her six firings and flares wide in the gap. By the end of this lesson you will be able to:

• Explain what "a distribution over functions" means and how the RBF kernel encodes "close temperatures give similar hardness"
• Compute the GP posterior mean and its predictive uncertainty from the kernel matrix, in a few lines of base R
• Read a GP band: why it pinches to roughly the noise level at a data point, flares in gaps, and returns to the prior far away
• Tune the lengthscale with the log marginal likelihood, and spot the overconfidence a wrong lengthscale causes

Prerequisites: the kernel trick lesson (the RBF kernel as a similarity score; here it takes on a second role), ordinary linear regression, the normal distribution (mean, sd, and what a 95% interval is), and the bias-variance trade-off from the ML Workflow course.

The interactive below is the whole lesson in miniature: a GP fit to six points, its 95% band pinching at the data and flaring between and beyond it. Toggle the lengthscale and watch the fit change character; we will build every part of this picture from scratch.

A prediction without a confession

Every regression tool you have met so far hands back one curve. Let us see what that costs Asha. Each lesson runs in a fresh R session, so we start by typing in her lab notebook: six firings, six hardness readings.

Hardness climbs to 584 at 1000 degrees, then is already falling by 1140. The peak Asha wants is probably somewhere inside the gap she never fired. Fit the standard tool, a quadratic regression (one parabola through the points), and ask it about three temperatures: 970 (right next to a firing), 1070 (the middle of the gap), and 1250 (past her hottest firing, into territory the kiln has never seen).

Look closely at the interval widths. At 970, where a real firing sits 30 degrees away, the interval is about 42 HV wide. At 1070, the centre of a 140 degree hole in the data, it is about 44 HV wide. Nearly identical. The parabola is just as confident where Asha has no evidence as where she has plenty, because its confidence comes from trusting the parabola shape everywhere. And at 1250 it calmly announces that hardness collapses to 418, a strong claim about a region it has never seen.

The shaded strip is the no-data gap, and the curve sails through it without blinking. What Asha needs is a model whose uncertainty is local: small where firings exist, large where they do not. That requires a different starting point entirely.