Sensitivity Analysis and Placebo Tests

Across this course you have built an arsenal for pulling a causal effect out of messy observational data: matching and propensity scores, inverse-probability weighting, difference-in-differences, regression discontinuity, instrumental variables. Every one of them shares a quiet catch. They strip out the confounding you can measure and put in a model, and none of them can rule out a confounder you never measured.

Take a real-sounding case. FreshCart, an online grocery, notices that customers who joined its paid membership, FreshCart Plus, are still active a year later far more often than customers who did not. It is tempting to announce that Plus causes loyalty. But maybe the keenest shoppers, the ones who love ordering groceries by app, both sign up for Plus and stick around regardless. That keenness is a confounder, and nobody wrote it down.

You cannot prove it is absent. So this lesson does the honest thing instead: it measures how fragile your conclusion is. By the end you will be able to:

• Explain why every estimate in this course rests on an untestable no-unmeasured-confounding assumption
• Compute an E-value: how strong a hidden confounder would have to be to explain your result away
• Read Rosenbaum bounds, the matched-design cousin of the E-value
• Run placebo tests, and recognise a failed one as proof your design is picking up bias

Prerequisites: Lessons 1 to 3 of this course (confounding and potential outcomes, propensity-score matching and IPW, and difference-in-differences); you can fit lm and glm and read a risk ratio and a p-value.

One assumption holds up the whole course

Look back at what each method actually did. Matching and IPW balanced the treated and control groups on the covariates you fed them. Difference-in-differences leaned on a control group with parallel trends. Regression discontinuity trusted that units just above and below a cutoff were otherwise alike. Different machinery, one shared load-bearing belief: once you account for what you measured, who ended up treated is as good as random.

That belief has a name, ignorability (also called no unmeasured confounding). Writing \(Y^1\) and \(Y^0\) for the outcome a unit would have with and without treatment \(T\), and \(X\) for the covariates you measured, ignorability says

\[ \{Y^0, Y^1\} \perp\!\!\!\perp T \mid X. \]

In words: among customers with the same measured covariates \(X\), which of them joined Plus is unrelated to how they would have behaved either way. If that holds, the leftover difference is the causal effect. If a confounder sits outside \(X\), it does not hold, and every estimate above is off by an unknown amount.