OLS Regression from Scratch
Priya runs a small iced-coffee cart outside a train station. On warm days she sells out; on cool days she carries stock home. For twelve days she wrote down two numbers: the day's high temperature and how many iced coffees she sold. She wants one honest rule, a straight line, that turns tomorrow's forecast into a guess for the day's sales.
There are infinitely many lines she could draw through her data. This lesson is about the single method that picks the best one: ordinary least squares, or OLS. By the end you will be able to:
- Say exactly what makes one line "best", and define a residual
- Compute that line two ways by hand (a formula, then the matrix normal equations) and confirm R's
lm()gets the identical answer - Read the line's slope and intercept, and judge the fit with R-squared
Prerequisites: you can run an R code block, read a scatterplot, and know what an average is. No calculus or matrix algebra assumed; we build those pieces up as we reach them.
The widget below is the whole lesson in miniature, running on Priya's real twelve days. Drag the slope and intercept: every point drops a red square onto the line, and the square's area is that day's error. Your goal, and OLS's goal, is to make the total red area as small as it can be. Press "Snap to least squares" to see where the math lands.
A line is a prediction machine
Here are Priya's twelve days. Each dot is one day: its temperature along the bottom, the cups she sold up the side. The dots climb from lower-left to upper-right, so warmer really does seem to mean more cups.
A straight line through that cloud is a rule for turning any temperature into a predicted number of cups. We write it
\[ \hat{y} = b_0 + b_1 x \]
Reading it in Priya's words: \(x\) is the day's temperature (the predictor), \(\hat{y}\) (said "y-hat") is the line's predicted cups, \(b_0\) is the intercept (the predicted cups at a temperature of zero, where the line crosses the vertical axis), and \(b_1\) is the slope (how many extra cups the line adds for each one-degree rise). The hat on \(\hat{y}\) matters: it is the line's guess, not the actual cups Priya sold.
Every choice of \(b_0\) and \(b_1\) is a different line, a different rule. Our whole job is to choose those two numbers well. First, let us keep Priya's data in R so we can work with it for the rest of the lesson. Each lesson starts a fresh R session, so we build the twelve days right here (run this first).
Every prediction has a residual
No straight line will pass through all twelve dots, so on most days the line's guess is a little off. That gap, for a single day, is the residual: the actual cups minus the predicted cups.
\[ e_i = y_i - \hat{y}_i \]
Here \(y_i\) is the cups Priya really sold on day \(i\), \(\hat{y}_i\) is the line's prediction for that same day, and \(e_i\) is the leftover. A residual has a sign. Take the least-squares line we are about to find: on the 20-degree day Priya sold 42 cups while the line predicts 39.69, so the residual is \(42 - 39.69 = +2.31\); she beat the line. On the 24-degree day she sold 44 while the line predicts 47.47, a residual of \(44 - 47.47 = -3.47\); the line over-guessed.
In the widget, each orange stick is one day's residual: the vertical distance from its dot up or down to the line. Drag the line and watch every stick grow and shrink at once. There is no line that kills them all; push one dot's residual to zero and others open up.
Add up the squared misses
To compare two whole lines we need a single number for how badly each fits, built from all twelve residuals. The obvious idea, adding the raw residuals, fails: positives and negatives cancel. For the best line Priya's residuals sum to exactly zero, yet that line is clearly not perfect.
The fix is to square each residual before adding. Squaring drops the sign (every term is now positive, so nothing cancels) and it punishes big misses far more than small ones. That gives the sum of squared errors:
\[ \text{SSE} = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \]
where \(n\) is the number of days (twelve here) and \(\sum\) means "add this up over every day". Watch the punishment at work: that tidy \(+2.31\) miss contributes \(2.31^2 \approx 5.3\) to the SSE, while the ugly \(-3.47\) miss contributes \(3.47^2 \approx 12.0\), more than double the damage even though the miss itself is only about 50% larger. In the widget, the SSE is literally the total area of all the red squares.
Ordinary least squares is the rule that picks the one line, the one pair \((b_0, b_1)\), that makes this total area as small as it can possibly be. Drag the sliders to shrink the SSE by hand, then press "Snap to least squares" to jump exactly to the minimum.
Squaring, not summing
On two days, a line's residuals are \(+8\) cups and \(-2\) cups. How much do these two days add to the sum of squared errors?
You do not have to drag: the closed form
Dragging finds the best line by feel. But with a single predictor there is an exact formula that lands on it directly, no searching. The slope is
\[ b_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2} \]
and once you have the slope, the intercept follows:
\[ b_0 = \bar{y} - b_1 \bar{x} \]
Every symbol is something you can compute from the data: \(\bar{x}\) is the mean temperature, \(\bar{y}\) is the mean cups, and the sums run over all twelve days. The slope has a clean meaning. The numerator measures how temperature and cups move together (on days hotter than average, is she also above average on cups?), and the denominator measures how much temperature alone spreads out. So the slope is co-movement per unit of temperature spread. The intercept formula then just slides the line up or down so it passes through the average day \((\bar{x}, \bar{y})\).
Let us run the formulas on Priya's numbers.
By hand, then: the intercept is about 0.82 and the slope is about 1.94 cups per degree. Hold on to those two numbers.
The normal equations
Priya has one predictor. Real models have many (temperature, day of week, a holiday flag). The formula above does not stretch past one predictor, but a single matrix expression does, and it is the engine inside every regression tool. Stack the data:
\[ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\beta} \]
Here \(\mathbf{y}\) is the column of actual cups, \(\boldsymbol{\beta}\) is the column of coefficients we want (intercept, then slope), and \(\mathbf{X}\) is the design matrix: one row per day, a first column of all 1s (that column pairs with the intercept) and then one column per predictor (here, temperature). The least-squares coefficients are given by the normal equations:
\[ \hat{\boldsymbol{\beta}} = (\mathbf{X}^{\top} \mathbf{X})^{-1}\,\mathbf{X}^{\top} \mathbf{y} \]
You reach this by writing the SSE as a function of \(\boldsymbol{\beta}\), taking its derivative, and setting it to zero (the calculus is optional here; the point is that minimizing squared error has one exact solution, not a search). Translated to R almost symbol for symbol:
The same 0.818 and 1.944, to the last digit. The drag, the formula, and the matrix all agree, because they are three routes to the one line that minimizes SSE. (This works whenever \(\mathbf{X}^{\top}\mathbf{X}\) can be inverted; it breaks only when two predictors are near-copies of each other, the multicollinearity trap you will meet in a later lesson.)
Let lm() do it in one line
You have now found Priya's line three ways. In practice you would never grind through any of that: R's lm() function ("linear model") solves the normal equations for you. It takes a formula, written response ~ predictor and read as "cups explained by temp". Fill in the blank so the model predicts cups from temperature.
Show answer
fit <- lm(cups ~ temp, data = coffee)
coef(fit)
#> (Intercept) temp
#> 0.8180113 1.9437148What the two numbers mean
coef(fit) gave an intercept of 0.82 and a slope of 1.94. Read them in Priya's world:
- Slope 1.94: each extra degree of temperature is associated with about 1.94 more cups sold, close to two cups per degree. This is the number Priya actually cares about.
- Intercept 0.82: the line's predicted cups at a temperature of zero. But zero degrees is nowhere near her data (her days run 15 to 31 degrees), so this is only the mathematical anchor where the line crosses the axis, not a real forecast.
The full summary even flags the intercept as untrustworthy: its p-value is 0.78, meaning it is statistically indistinguishable from zero. The slope, by contrast, is rock solid (its p-value is 0.0000000154).
And the slope only speaks for the range Priya has actually seen. Predicting for a pleasant 25-degree day is fair; predicting for an imaginary 40-degree day is not.
The 25-degree prediction (about 49 cups) sits inside her data and is believable. The 40-degree prediction (about 79 cups) is pure extrapolation: no day that hot exists in the data, and a real 40-degree heatwave might empty the platform instead of packing it. A line only speaks for the range it has seen.
How good is the fit? R-squared
The slope tells you the direction and size of the effect. R-squared tells you how tightly the dots hug the line. Start from the dumbest possible model: ignore temperature and always guess the average, \(\bar{y}\). Its total squared error is the total sum of squares:
\[ \text{SST} = \sum_{i=1}^{n} (y_i - \bar{y})^2 \]
The regression line does better, leaving behind its smaller SSE. R-squared is the fraction of that baseline error the line manages to remove:
\[ R^2 = 1 - \frac{\text{SSE}}{\text{SST}} \]
If the line explained nothing, SSE would equal SST and \(R^2 = 0\). If it passed through every dot, SSE would be 0 and \(R^2 = 1\). For Priya:
Temperature explains about 96% of the day-to-day swing in Priya's sales, a genuinely strong fit. Two footnotes worth carrying forward:
- With a single predictor, \(R^2\) is exactly the square of the correlation \(r\) (a single number from -1 to 1 that measures how tightly two things move together, where 1 is a perfect upward line). Here \(r = 0.982\) and \(0.982^2 = 0.964\), the same number.
- \(R^2\) never falls when you add a predictor, even a useless one, so a high \(R^2\) is not proof the model is right. Whether these residuals are actually well behaved is the entire subject of the next lesson.
Read Priya's model
Priya's model has a slope of 1.94 and an R-squared of 0.96. Which reading is correct?
References
- An Introduction to Statistical Learning, Chapter 3 (free PDF) - the friendliest full treatment of simple and multiple linear regression, with worked R.
- The Elements of Statistical Learning, Chapter 3 (free PDF) - the rigorous version, including the least-squares geometry and the normal equations.
- R documentation for lm() (stats package) - the exact function you used, its formula interface, and everything it returns.
- Ordinary least squares (Wikipedia) - a careful derivation of the normal equations and the assumptions behind them.
Lesson 1 complete
You built ordinary least squares from the ground up. A line predicts, a residual is its miss, and the best line is the one that minimizes the total squared miss. You found Priya's line three ways that all agreed (by dragging, by the closed-form formula, and by the matrix normal equations), then let lm() do it in a single line. You read the slope (about two cups per degree), treated the intercept with suspicion, and used R-squared to see that temperature explains about 96% of her sales.
That last number is seductive. A line can fit beautifully and still be quietly wrong, because lm() hands you a slope and a p-value whether or not the data earned them. Next, in Regression Assumptions and Residuals, you will meet the four promises every regression leans on, and learn to read a residual plot to check whether Priya's model actually kept them.