Baseline Imbalance in Peer Review

Baseline imbalance is when the groups you compare already differ on some characteristic before the treatment or exposure could act, so part of the outcome gap might be that head start rather than the effect you are testing. Whether it matters depends entirely on whether the imbalanced characteristic predicts the outcome, so the honest answer is to check the ones that do and report what adjusting for them changes.

What the reviewer wrote

The two arms appear to differ at baseline on several characteristics. The authors may wish to confirm that these differences do not account for the reported effect.

The groups are not comparable at baseline. Age and disease severity both differ, and the analysis makes no adjustment for either.

The manuscript is well written and the question is timely. My concern is with Table 1: the intervention arm is younger and has more men, and since both are associated with the outcome, I am not persuaded that the unadjusted difference in Table 2 reflects the intervention rather than the composition of the arms.

What they actually mean

The reviewer is saying the groups were not on equal footing before the thing you are studying, so a difference in the outcome could reflect who ended up in each group rather than the treatment itself. What they want is for you to show the groups were similar on outcome-relevant characteristics, or to adjust for the ones that were not and report what that adjustment does.

What they are not asking for is a significance test on every row of Table 1. That reading is common and it is backwards: as the practice at the end shows, a significant baseline difference is not the signal that you must adjust, and a non-significant one is not permission to skip it. The trigger for adjustment is whether a characteristic predicts the outcome, not the p-value attached to its imbalance. One more distinction settles which version of the problem you have. In an observational study the groups were never randomised, so baseline imbalance is confounding and the response is the adjustment argument below. In a randomised trial the imbalance arose by chance, and the design changes what you owe the reviewer.

Why they are asking

If a characteristic differs between the groups and it predicts the outcome, then part of the outcome gap is the head start, and reading the whole gap as the treatment effect overstates it, understates it, or in the worst case flips its sign. The size of the distortion tracks how strongly the imbalanced characteristic predicts the outcome. A large imbalance on something irrelevant to the outcome does nothing at all, whereas a modest imbalance on a strong predictor can move the estimate a long way, which is why the check has two parts rather than one: is the characteristic imbalanced, and does it predict the outcome.

The design decides how worried to be. Randomisation balances characteristics between arms in expectation, so any observed imbalance is a chance event and the design itself answers most of the objection (Senn, 1994, Statistics in Medicine). An observational comparison carries no such guarantee, and there the reviewer is right to ask before any causal reading is allowed. The mechanics of adjusting a group comparison for a covariate are covered in ANCOVA in R; this chapter is about deciding whether the imbalance matters and how to report it.

How to check it

Take a finding from mtcars: cars with a straight engine (vs = 1) are far more fuel-efficient than those with a V-shaped engine (vs = 0). A reviewer objects that the two groups of cars were never comparable to begin with. Start with the unadjusted difference.

Straight-engine cars average 7.94 mpg more, and the difference is clearly significant. Now check the baseline. The way journals present balance is a Table 1 carrying a standardized mean difference for each characteristic, which is the difference in group means divided by the pooled standard deviation, so it does not shrink or grow with the sample size the way a p-value does (Austin, 2009, Statistics in Medicine). It is a few lines of base R.

Each SMD is the straight-engine mean minus the V-shaped mean, in pooled standard deviations. Displacement, horsepower and weight are all past 1.3 in absolute value, so the groups are markedly different at baseline. The convention is to treat anything above about 0.1 as imbalanced (Austin, 2009), which flags every row here, but that threshold is a convention rather than a law. Read a large SMD as a prompt to ask whether the variable predicts the outcome, not as a verdict, because that second question is what decides whether the imbalance bites.

What to do about it

You are fine

The imbalance sits on a characteristic that does not drive the outcome, or the study was randomised. Take the number of forward gears. It is imbalanced across the groups, at an SMD of 0.42, but only moderately related to fuel economy, correlating with mpg at 0.48. Add it to the model.

The engine-shape estimate moves only from 7.94 to 7.06 mpg and stays significant, so the gear imbalance is not what produced the result. Report the balance table and this adjusted model side by side so the reviewer can see the estimate held. In a randomised trial the argument is shorter still: randomisation balances characteristics in expectation, so a baseline difference is a chance event, and CONSORT asks for the balance table (item 15) rather than a significance test on it (Senn, 1994). Whether every imbalanced characteristic behaves as gently as gear depends on how strongly it predicts the outcome, which is the next case.

It is fixable

A strongly prognostic characteristic is imbalanced. Displacement is imbalanced at an SMD of -2.04 and it predicts fuel economy hard, correlating with mpg at -0.85. Adjust for it.

Once displacement is in the model, the engine-shape advantage falls from 7.94 mpg to 1.50 and is no longer distinguishable from zero, because most of the apparent gap was that the straight-engine cars simply had smaller engines. The fix is to report this rather than bury it. Present the unadjusted and adjusted estimates together and reword the claim so it credits engine size, not engine shape. In an observational study STROBE item 16(a) asks for exactly that pairing; in a randomised trial the adjustment should have been pre-specified on prognostic grounds rather than chosen because the covariate came out imbalanced (EMA guideline on adjustment for baseline covariates, 2015; Assmann et al., 2000, Lancet), a point the practice returns to. When the groups barely overlap on the covariate, a single regression term is not enough and matching or a propensity score is the better tool.

It is a real problem

Sometimes the groups differ so much on a key characteristic that they share almost no common range, and then no adjustment can compare like with like, because the model has to extrapolate into a region where one group has no data. To illustrate with numbers, imagine a two-arm study where the treated patients have baseline severity scores of 8 to 10 and the controls score 1 to 4, with almost no overlap. A regression that adjusts for severity will still print a treatment coefficient, but that number rests on assuming the severity-to-outcome relationship holds where you never observed both arms, and it can swing from something like +2.1 to -0.4 as you change the functional form. No reanalysis rescues this, because the data do not contain the comparison the reviewer wants made. The honest paths are to restrict the analysis to the overlapping range and report the narrower question you can actually answer, or to state in the Limitations that the arms are not comparable and the contrast is not identifiable from these data.

How to word your response

If you are fine

The reviewer notes that the groups differ at baseline. We now present standardized mean differences for all baseline characteristics (Table 1). The number of forward gears differs between the groups but is only weakly related to the outcome, and adding it to the model leaves the group estimate essentially unchanged, from 7.94 to 7.06 mpg. We report the adjusted model alongside the unadjusted one (Results, page X) so that the stability is visible to the reader.

If it is fixable

We thank the reviewer for this point. The two groups differ substantially in engine displacement at baseline, and displacement is a strong predictor of fuel economy. After adjusting for it, the difference attributed to engine configuration falls from 7.94 to 1.50 mpg and is no longer statistically distinguishable from zero (p = 0.37). We have revised the Results (page X) to present the unadjusted and adjusted estimates together, and have reworded the conclusion so that it no longer credits engine shape with an effect that is carried by engine size.

If it is a real problem

We agree that the groups are not comparable on baseline severity, which ranges from 8 to 10 in the treated arm and 1 to 4 in the controls, with almost no overlap. Because no region of common support exists, a covariate-adjusted estimate would rest on extrapolation rather than data, and we do not report one as though it were reliable. We have restricted the primary analysis to the overlapping range (Methods, page X), reported how many participants fall outside it, and stated in the Limitations that the full-sample contrast is not identifiable from these data.

Practice

A reviewer writes: "Dose is strongly associated with tooth length and is unlikely to be balanced across the two supplement groups. The unadjusted comparison of OJ against VC is therefore probably biased by dose, and the authors should adjust for it." You run the check:

Which of the three outcomes applies, and what do you write back?