Measuring how optimistic your model is
Today let's understand why the R-squared you compute on the same data you fitted your model on is almost always too good, and how to measure exactly how much too good it is.
Let's say you are a researcher with data on 60 patients. You measured 10 candidate predictor variables on each one, x1 through x10, but only two of them, x1 and x2, actually move the outcome y you care about. You fit a regression using just those two predictors and score it on the same 60 patients it was built on. The R-squared comes back at 0.085.
Now correct that number honestly, for exactly how much it overstates the model's real performance. Once that correction is applied, the R-squared drops to -0.01, telling you the model barely predicts anything about a patient it has not seen.
Look at those two numbers side by side.
That is the whole gap this lesson measures. Same model, same 60 patients, but the score changes completely depending on whether it is scored on the data that built it or corrected for how much that data overstates it.
The score you get from your own data is too good
Start with the usual way to judge how well a regression fits: R-squared. It is the share of the variation in the outcome that the model's predictions account for. An R-squared of 1 means the model explains all of that variation, an R-squared of 0 means the model does no better than guessing the average outcome every time, and R-squared can drop below 0 once the model's predictions are further off than that plain average would be.
Now build the 60 patients. Each one carries 10 candidate predictor columns, x1 through x10, drawn as plain random noise. Only x1 and x2 actually move the outcome y, with coefficients 1.5 and -1.0. The other 8 columns carry no real relationship to y at all, they exist because real datasets arrive full of columns like that, plausible-looking but empty.
Fit a regression of y on x1 and x2, the two predictors that genuinely matter here, and check its R-squared.
The coefficients land close to the true 1.5 and -1.0, so the model found roughly the right relationship. Call 0.085 the apparent R-squared: the honest model's score, computed on the same 60 patients it was fitted on.