Matching and the Propensity Score
A hospital offers an optional cardiac-rehab program to patients recovering from a heart attack. Three months later, the patients who enrolled walk 52 metres further in six minutes than the patients who did not. A press release writes itself: rehab adds 52 metres. Except it did not. The patients who signed up were fitter and more motivated to begin with, so they would have recovered better even with no program at all. Part of that 52-metre gap is the rehab; part is just who chose to enroll.
This is the central problem of learning from data you did not get to randomize. When people pick their own treatment, the treated and untreated groups differ for reasons that have nothing to do with the treatment. This lesson gives you the first and most intuitive tool for fixing it: estimate how likely each person was to be treated (their propensity score), pair each treated person with an untreated look-alike, and compare only fair matches. Toggle the panel below to feel the whole idea in miniature: two groups that start out mismatched, snapped into alignment.
By the end of this lesson you will be able to:
- Explain why a naive difference between a treated and an untreated group is biased when people self-select
- State what a causal effect actually is, using potential outcomes, and split the naive gap into a real effect plus selection bias
- Estimate a propensity score with
glm, match on it, and recover an effect a raw comparison gets badly wrong - Read the standardized mean difference to check that matching worked, and name what matching still cannot fix
Prerequisites: you can fit a logistic regression with glm(..., family = binomial) and read its predicted probabilities, and you know why a randomized experiment removes confounding.
You cannot just compare the two groups
Let us build the hospital's data so the trap is impossible to wave away. Four hundred patients; each has a baseline fitness at discharge (metres walked in six minutes) and either enrolled in rehab or did not. We plant a true rehab effect of exactly 25 metres, so we know the honest answer and can watch the naive comparison miss it. Crucially, fitter patients (higher baseline) are both more likely to enroll AND recover more on their own, which is the confounding.
The enrolled group gained 78.5 m on average, the others 26.4 m. Subtract, and you get the headline.
A 52.1-metre gap, more than double the 25 metres we actually built in. Why is it so inflated? Because we are not comparing rehab against no-rehab. We are comparing the kind of patient who enrolls against the kind who does not, and those two kinds of patient were different before rehab entered the picture. The next step draws exactly why that ruins the comparison.
Confounding, drawn
A picture makes the problem precise. Call the treatment X (enrolled in rehab), the outcome Y (three-month gain), and baseline fitness Z. In our study, baseline fitness is a confounder: it points an arrow at X (fitter patients enroll more) and an arrow at Y (fitter patients recover more), while X also affects Y (rehab genuinely helps). Because Z feeds both X and Y, some of the X-to-Y association you observe is really Z leaking through a side route, the backdoor path X ← Z → Y.
Switch the widget to confounder and read the verdict: when a variable causes both the treatment and the outcome, you must control for it, or the effect you read off is wrong. (The other two shapes, collider and mediator, are traps in the opposite direction, and later lessons return to them. For now, baseline fitness is a clean confounder.)
Matching is one way to "control for Z": instead of comparing all treated to all controls, we will compare treated and control patients who have the same baseline fitness, closing the backdoor.
The comparison you wish you had
To say the naive gap is "wrong" we need to say what "right" would be. The clean way is potential outcomes. For each patient \(i\), imagine two parallel worlds: \(Y_i(1)\) is the gain they would record if they enrolled in rehab, and \(Y_i(0)\) is the gain the same patient would record if they did not. That patient's true rehab effect is the difference between their own two worlds,
\[ \tau_i = Y_i(1) - Y_i(0). \]
Here is the catch that defines the whole field, the fundamental problem of causal inference: a patient either enrolls or does not, so you only ever observe one of \(Y_i(1)\) and \(Y_i(0)\). The other is a counterfactual you can never measure.
| Patient | Enrolled? | Gain if enrolled, \(Y(1)\) | Gain if not, \(Y(0)\) | Effect \(\tau_i\) |
|---|---|---|---|---|
| Anaya | yes | 74 m (observed) | ? never seen | ? |
| Ben | no | ? never seen | 22 m (observed) | ? |
Since we cannot get an individual's two worlds, we aim for an average. The average treatment effect on the treated (ATT) is what rehab did, on average, for the patients who actually took it, with \(T = 1\) meaning treated:
\[ \mathrm{ATT} = E\big[\,Y(1) - Y(0) \mid T = 1\,\big]. \]
Now watch what the naive difference of observed group means actually equals. Adding and subtracting the term \(E[Y(0)\mid T=1]\) (how the treated would have done untreated) splits it cleanly:
\[ \underbrace{E[Y \mid T{=}1] - E[Y \mid T{=}0]}_{\text{naive 52.1 m}} = \underbrace{E[Y(1){-}Y(0)\mid T{=}1]}_{\text{ATT, the real effect}} + \underbrace{E[Y(0)\mid T{=}1] - E[Y(0)\mid T{=}0]}_{\text{selection bias}}. \]
The second term is the whole trouble: it asks whether the treated and control groups would have differed even with no treatment. In our study they would, because the treated started fitter, so \(E[Y(0)\mid T{=}1] > E[Y(0)\mid T{=}0]\) and the bias is positive. That extra 27 metres of "bias" is exactly what pushed 25 up to 52. Kill the selection bias and the naive number collapses back onto the ATT.
Why the 52 metres is not the effect
In our data the enrolled patients had a mean baseline of 381 m versus 322 m for the others, and the raw outcome gap was 52.1 m against a true effect of 25 m. Which statement correctly explains the gap?
The propensity score
Matching patient to patient on baseline fitness works when you have one confounder. Real studies have dozens (age, prior surgeries, medication, distance from the clinic), and you cannot find an exact twin across all of them. Rosenbaum and Rubin's insight was to collapse every confounder into a single number: the propensity score, the probability that a patient with covariates \(x\) ends up treated,
\[ e(x) = P(T = 1 \mid X = x). \]
You already know how to estimate a probability from predictors: a logistic regression. Fit the treatment (not the outcome) on the confounders, and read off each patient's fitted probability of enrolling.
Read row 1: a very fit patient (419 m) had a 0.76 chance of enrolling but happened not to, an unusually good control to compare a treated patient against. The magic is the balancing property: among patients who share the same propensity score, the treated and the untreated have, on average, the same distribution of the original confounders. So comparing a treated and a control patient with equal scores is like comparing two people plucked from the same mini-randomized experiment. One number stands in for the whole tangle of confounders.
Match on the score, and the imbalance collapses
The simplest recipe is nearest-neighbour matching: walk through each treated patient, and pair them with the control whose propensity score is closest. That throws away nothing about the treated group (so the estimate targets the ATT) and reuses the controls that actually resemble treated patients. It only works where the two groups overlap, the region of "common support" where both treated and control patients exist; a treated patient so extreme that no control comes close cannot be matched honestly, and you should not pretend otherwise.
How do you check that matching balanced a confounder? Compare its group means on a scale that ignores units, the standardized mean difference (SMD). For a covariate with treated mean \(\bar{X}_t\), control mean \(\bar{X}_c\), and group variances \(s_t^2, s_c^2\),
\[ d = \frac{\bar{X}_t - \bar{X}_c}{\sqrt{\tfrac{1}{2}\left(s_t^2 + s_c^2\right)}}. \]
It is the gap in means measured in standard deviations. A common rule of thumb: \(|d| < 0.1\) is good balance, meaning the groups are close enough on that covariate to compare fairly. Toggle the widget between before and after matching: the mismatched treated and control piles slide into alignment, and the SMD printed underneath drops from badly imbalanced toward zero.
Recover the real effect
Now do it on our study. The propensity score study$ps already exists from two steps ago. Pair each treated patient with the nearest-propensity control by filling in the function that returns the index of the smallest distance, then average the paired differences to estimate the ATT.
Show answer
trt <- which(study$rehab == 1)
ctrl <- which(study$rehab == 0)
match_id <- sapply(trt, function(i) ctrl[which.min(abs(study$ps[ctrl] - study$ps[i]))])
att <- mean(study$improve[trt] - study$improve[match_id])
naive <- mean(study$improve[trt]) - mean(study$improve[ctrl])
round(c(true = 25, naive = naive, matched = att), 1)
#> true naive matched
#> 25.0 52.1 27.0
# did matching actually balance baseline fitness?
smd <- function(a, b) (mean(a) - mean(b)) / sqrt((var(a) + var(b)) / 2)
round(c(before = smd(study$baseline[trt], study$baseline[ctrl]),
after = smd(study$baseline[trt], study$baseline[match_id])), 2)
#> before after
#> 0.93 0.04Reading the balance number
After matching, the standardized mean difference on baseline fitness fell from 0.93 to 0.04. What can you legitimately conclude?
What matching cannot do
Matching bought us a fair comparison on baseline fitness. It is not a truth machine, and two assumptions do the heavy lifting.
The second assumption is overlap (positivity): every kind of patient must have some real chance of being in either group. A treated patient whose propensity score is 0.99, with no control anywhere near, has no honest match, and forcing one invents a comparison the data cannot support. Always inspect the score distributions before trusting the estimate.
In practice you would not hand-roll the loop above. The MatchIt package does the propensity model, the matching, and the balance diagnostics in a couple of lines (this needs a full R install, so read it, do not run it here):
library(MatchIt)
m <- matchit(rehab ~ baseline, data = study, method = "nearest")
summary(m) # standardized mean differences, before and after
matched_data <- match.data(m) # the paired sample, ready for an outcome model
One more sanity check worth knowing: matching is not the only adjustment. Regressing the outcome on treatment and the confounder, lm(improve ~ rehab + baseline), recovers a rehab coefficient of 25.6 m here, agreeing with matching. When two different adjustments land in the same place, you can trust the answer more. When they disagree, something (often overlap or an unmeasured confounder) deserves a closer look.
References
Five authoritative places to take this further:
- Austin (2011), An Introduction to Propensity Score Methods (PMC) - the clearest applied primer; walks through matching, weighting, stratification, and adjustment.
- Stuart (2010), Matching Methods for Causal Inference: A Review - the authoritative survey of where nearest-neighbour matching sits among the alternatives.
- MatchIt: Getting Started (Ho, Imai, King and Stuart) - the R package that runs everything in this lesson, plus balance diagnostics, in a few lines.
- Hernan and Robins, Causal Inference: What If (free PDF) - the rigorous treatment of confounding, exchangeability, and positivity, the assumptions matching leans on.
- Cunningham, Causal Inference: The Mixtape - Matching and Subclassification - a friendly, worked companion with code.
Lesson 1 complete
You saw a 52-metre headline dissolve into 25 metres of real effect once you stopped comparing the kind of patient who enrolls with the kind who does not. You defined a causal effect with potential outcomes, split the naive gap into the ATT plus selection bias, estimated a propensity score with glm, matched treated patients to their control look-alikes, and confirmed with the standardized mean difference that baseline fitness was finally balanced.
Next, Lesson 2: Inverse-Probability Weighting and Doubly-Robust estimation. Instead of discarding unmatched controls, you will reweight every patient by their propensity to rebuild a pseudo-randomized sample, meet the overlap requirement in a sharper form, and learn the doubly-robust estimator that still works if either the propensity model or the outcome model is right, a genuine second chance at an unbiased answer.