Selection Bias in Peer Review

Selection bias is when the units in your sample are not a fair stand-in for the population you set out to describe, because whatever decided who got into the sample, or who stayed in it, was itself tied to the outcome you are measuring. It distorts a simple average as readily as a group comparison, and unlike confounding you often cannot remove it by adding a variable to the model, since the units that would move the estimate are the ones missing from your data.

What the reviewer wrote

The authors do not report the response rate, nor whether the participants who responded differ from those who did not. A comment on the potential for selection bias would strengthen the paper.

Only about 40% of those approached took part. At that level of non-response the sample cannot be assumed representative, and the estimates may be biased.

The analysis is careful and the manuscript reads well. My one substantive concern is recruitment: participants came through a single specialist clinic, and patients referred there are unlikely to resemble the wider population the abstract generalises to. I would want to know who was eligible but never entered the sample before I accept the headline figure.

What they actually mean

The reviewer is asking whether your sample represents the population you want your numbers to speak for. A study observes a slice of the world and then reports as though the slice were the whole, which only holds when the slice was carved out for reasons unrelated to what you are measuring. When the reason someone entered your data, or stayed in it, is connected to their outcome, the slice is tilted and every estimate drawn from it inherits the tilt.

This is not the same objection as confounding, and reading it as one sends you down the wrong road. Confounding is a difference between the groups you compare on a variable you measured and can adjust for. Selection bias is about who is in the dataset at all, so the people who would move the estimate are absent and there is nothing in the model to adjust. The other common misreading treats the response rate as the verdict: a high rate does not prove the sample is representative, and a low one does not prove it is biased, because the question is whether entering or leaving the sample is related to the outcome rather than how many people took part.

Why they are asking

If being in the sample is tied to the outcome, the sample no longer centres on the population value, and reporting the sample number as the population number is off by however strongly selection and outcome are linked. The direction is not fixed. When the people more likely to respond tend to have higher outcomes, the estimate runs high; when the sicker patients are the ones who drop out before the final visit, it runs low. How large the distortion is depends on two things at once, so the check has two parts: is entry into the sample related to some characteristic, and does that characteristic relate to the outcome.

One distinction saves a lot of needless worry. Selecting on the exposure, or on a cause of the exposure, does not bias the exposure-outcome association inside a cohort; selecting on the outcome, or on a variable that both the exposure and the outcome influence, does (Hernán, Hernández-Díaz and Robins, 2004, Epidemiology). So a study that enrolled people because of their treatment can still estimate the treatment effect cleanly, whereas one that enrolled them because of how they fared cannot. What a sample is and how bias enters it is covered in Populations, Samples and Bias in R; here the job is to work out whether the tilt is real and what to report.

How to check it

Suppose you want to report the average fuel economy of a class of cars, and the population you care about is all 32 cars in mtcars, whose true mean is 20.09 mpg. You do not observe all of them. A response mechanism decided which cars came back for testing, and heavier cars were less likely to return. Start by comparing the sample you ended up with against the population it is meant to describe.

The respondents average 22.07 mpg against a population value of 20.09, and the cars that never responded sit down at 18.35. A 47% response rate produced a sample that reads almost two miles per gallon too optimistic. There is no response-rate cutoff that certifies a sample as safe, and the guidelines that once demanded 60% or 70% have largely been dropped, because a rate is only a proxy for the thing that matters, which is whether responders and non-responders differ on the outcome (Groves, 2006, Public Opinion Quarterly). So look at how the two groups differ on the characteristics you hold for both.

The respondents are lighter by nearly a ton on average, with less power and smaller engines. Weight is the strongest single predictor of fuel economy here, correlating with mpg at -0.87, so a sample skewed toward light cars is skewed toward high mileage. Non-response bends an estimate only when the sample is tilted on something the outcome depends on, and this one is. The check has a hard boundary worth naming: you can only compare respondents and non-respondents on variables you hold for both, so a characteristic recorded only on the people who stayed in says nothing about the ones who left.

What to do about it

You are fine

The selection is unrelated to the outcome, or related only through a variable the outcome does not depend on. Non-response can be substantial and still leave the estimate alone, as long as who responded was not tied to what you are measuring. Take the same cars, but let response be a coin flip that ignores every characteristic of the car.

At a 44% response rate, barely higher than the biased case, the respondent mean lands at 20.34 against a population 20.09. That quarter-mpg gap is chance rather than tilt, because nothing about a car changed its odds of responding. This is the missing-completely-at-random case from the missing-data literature, where the complete-case estimate is unbiased however many values are absent (Little and Rubin, 2019, Statistical Analysis with Missing Data). To show a reviewer you are in this position, report the response rate, then present the respondent-versus-non-respondent comparison from the check and let the near-identical covariate means carry the argument.

It is fixable

You have the variable that drove selection, and you have it for the whole frame, not just the responders. When selection depends on measured characteristics, you can weight each respondent by the inverse of their probability of responding, so the under-represented kinds of unit count for more and the sample is pulled back toward the population it came from. Fit a model for who responded, turn it into weights, and re-estimate.

The naive respondent mean of 22.07 falls to 20.77 once each car is weighted by the inverse of its estimated probability of responding, against a population value of 20.09. Weighting up-counts the heavy cars that were unlikely to come back, which undoes most of the skew on the one variable that drove selection. It does not land exactly on 20.09, because weighting removes only the part of the selection the measured variable explains, and an honest analysis reports the weighted estimate with a wider interval that reflects the uncertainty the weighting adds. The method and its standard errors are covered in Inverse Probability Weighting; when selection is thought to depend on the unobserved outcome rather than on measured covariates, a Heckman selection model is the tool reviewers in economics will expect. Report the corrected estimate alongside the naive one so the reader can see the direction and size of the correction.

It is a real problem

Selection is tied to the outcome itself, or to something you never measured on the people who left, and you have no data on them to weight or model your way back. Picture a prevalence study recruited through a specialist clinic, where patients arrive precisely because their condition is severe. The people with mild, undiagnosed disease are missing from the frame entirely, so the sample cannot be reweighted to include them: there is no record of them to weight, and severity, the thing that governed entry, is the outcome you are trying to estimate. A reported prevalence of, say, 38% might correspond to anything from 12% to 20% in the general population, and no reanalysis of the clinic data narrows that down, because the information needed sits in the people who never came. The honest path is to stop presenting the clinic figure as a population figure. Report it as what it is, a clinic-based estimate, state in the Limitations that the recruitment route selects on severity and biases the prevalence upward, and where you can, put numbers on the plausible range with a sensitivity analysis rather than leaving the direction of the bias for the reader to guess.

How to word your response

If you are fine

The reviewer asks about non-response and the potential for selection bias. Our response rate was 44%, and we have added a comparison of respondents and non-respondents on the characteristics recorded for the full eligible sample (Methods, page X); the two groups are closely matched on all of them, including the strongest predictors of the outcome. Because entry into the sample was not related to the outcome, the reported mean is unbiased despite the response rate, and we now state this explicitly alongside the comparison table.

If it is fixable

We thank the reviewer for raising this. Respondents did differ systematically from non-respondents, being lighter on average, and because that characteristic strongly predicts the outcome the unadjusted mean was optimistic. We have re-estimated the mean using inverse-probability-of-response weights built from the characteristics available for the whole sampling frame; the estimate falls from 22.07 to 20.77 once the under-represented units are up-weighted (Results, page X). We now report the weighted estimate as the primary result, with the unweighted value and the response model given alongside it.

If it is a real problem

We agree that recruitment through a specialist clinic selects patients on severity, and that severity is bound up with the prevalence we set out to estimate. Because we have no data on the eligible people who were never referred, no weighting or adjustment can recover the population figure from these data, and we do not present one as though it could. We have relabelled the estimate as a clinic-based prevalence throughout (Abstract and Results, page X), added a Limitations paragraph stating that the sampling route biases it upward, and included a sensitivity analysis giving the plausible range under stated assumptions about the unreferred population.

Practice

A reviewer writes: "Nearly a quarter of the ozone readings in your series are missing. A missingness rate this high makes the reported mean unreliable, and the authors should address the resulting selection bias." The daily air-quality series has ozone missing on many days, and you hold temperature, wind and solar radiation for every day, whether or not ozone was recorded. You run the check:

Which of the three outcomes applies, and what do you write back?