Fitting a curve without guessing its shape

Today let's understand how to fit a regression curve without guessing its shape ahead of time.

Vantage Analytics has experience and salary on file for 150 of its data scientists: years of experience on one axis, annual salary in thousands of dollars on the other. Here is what that relationship looks like.

Salary climbs fast in the first several years of experience, then the climb slows until it nearly flattens out. A straight line could not follow a bend shaped like that, which is exactly the shape a curve needs to pick up without anyone hand picking it in advance.

Why a straight line misses this bend

Vantage Analytics' numbers live in vantage_data: one row per data scientist, with their years of experience and their annual salary in thousands of dollars. Start the simplest way possible: fit a straight line through it with lm(), and see how well that line does.

RInteractive R
# Simulate 150 data scientists' experience and salary, then fit a straight line through them set.seed(42) n <- 150 experience <- round(runif(n, 1, 30), 1) salary <- round(55 + 42 * (1 - exp(-experience / 8)) + rnorm(n, 0, 4), 1) vantage_data <- data.frame(experience, salary) fit_line <- lm(salary ~ experience, data = vantage_data) summary(fit_line) #> #> Call: #> lm(formula = salary ~ experience, data = vantage_data) #> #> Residuals: #> Min 1Q Median 3Q Max #> -17.4829 -3.3823 0.1382 3.2235 12.1375 #> #> Coefficients: #> Estimate Std. Error t value Pr(>|t|) #> (Intercept) 70.87546 1.02515 69.14 <2e-16 *** #> experience 1.00622 0.05499 18.30 <2e-16 *** #> --- #> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 #> #> Residual standard error: 5.727 on 148 degrees of freedom #> Multiple R-squared: 0.6935, Adjusted R-squared: 0.6914 #> F-statistic: 334.9 on 1 and 148 DF, p-value: < 2.2e-16

  

R-squared comes out at 0.6935, so the line explains about 69% of the variation in salary. Each extra year of experience adds about $1,006 to the predicted salary, since the slope, 1.00622, is in the same thousands-of-dollars unit as salary itself, starting from an intercept near $70,875.

That R-squared alone does not say whether the line is missing something systematic. For that, look at its residuals: the gap between each data scientist's actual salary and what the line predicts for them. Split experience into five equal-width bands, each spanning about six years, and average the residuals inside each one.

RInteractive R
# Average the line's residuals inside five equal-width bands of experience breaks <- c(1, 6.7, 12.5, 18.2, 24, 29.7) bands <- cut(vantage_data$experience, breaks = breaks, include.lowest = TRUE) round(tapply(resid(fit_line), bands, mean), 2) #> [1,6.7] (6.7,12.5] (12.5,18.2] (18.2,24] (24,29.7] #> -5.26 3.70 4.82 1.09 -3.57

  

Read that from left to right. In the lowest band the line overpredicts salary by 5.26 thousand dollars on average, a negative residual. Move into the next three bands and it underpredicts, by 3.70, then 4.82, then 1.09. Then in the highest band it overpredicts again, by 3.57.

That is not noise. If the line were catching the real relationship, its residuals would scatter randomly above and below zero with no pattern tied to experience. Instead they run negative, then positive for a stretch, then negative again. That shape is exactly what you would see if the true relationship is concave, rising fast and then levelling off: a curve like that sits above its own straight-line approximation through the middle of the data, and below it at both ends.

The straight line is missing the bend, and its residuals are the proof.