Unplanned Subgroup Analyses in Peer Review

A reviewer who flags an unplanned subgroup analysis is not calling your subgroup result false. They are pointing out that you split the sample after seeing the data, tested inside the pieces, and reported the piece that reached significance, which is a reliable way to find effects that are not there. The answer has two parts: test the subgroup difference the correct way, with an interaction term, and be honest about whether you planned the split in advance.

The objection stings because it reads as an accusation of fishing, and sometimes that is exactly what it is. More often the reviewer is being careful rather than hostile, and the check takes a minute. Running the interaction test tells you whether the difference between subgroups is real, and that is what decides which version of the objection you are actually answering.

What the reviewer wrote

The same concern arrives in very different tones.

The subgroup finding is interesting, but it does not appear to have been pre-specified. Could the authors clarify whether this analysis was planned, and how many subgroups were tested in total?

You cannot claim the treatment works in one group just because it was significant there and not in the other. That is not a test of a difference. Report the interaction.

The methods are otherwise clear. I would note only that the effect in the older cohort is presented as a headline result, although the study lists a single primary analysis in the full sample and was not powered for age strata, so some comment on the status of this particular analysis would help the reader interpret it.

What they actually mean

Two separate worries hide inside this one comment, and they pull in different directions. The first is procedural: did you choose the subgroup before you saw the data, or after? A split you committed to in advance is a planned test, whereas a split you arrived at by trying several is one of many implicit comparisons, and its p-value no longer means what it appears to mean. The second worry is statistical, and it is the one authors most often act on wrongly: a result that is significant in one subgroup and not in another is not evidence that the effect differs between them. The reviewer is asking you to establish any difference with the correct test and to say plainly which analyses were planned. They are not asking you to delete the subgroup.

Why they are asking

Splitting a sample multiplies your chances to find something. With two or three subgroups and a few outcomes you have quietly run a dozen tests, and at least one tends to cross 0.05 by luck alone, the same multiplicity problem covered in Multiple Comparisons in Peer Review. Report only the test that landed and you have a finding assembled from noise and presented as a discovery about a particular group. The second failure is quieter and catches careful people. Comparing two within-subgroup p-values feels like comparing the two effects, but it is not: an effect can be significant in a large subgroup and non-significant in a smaller one purely because the smaller group has less power, even when the true effect is the same size in both (Gelman and Stern, 2006). A reviewer who has watched subgroup claims fail to replicate is asking you to rule that artifact out before you build on it.

How to check it

A subgroup analysis takes one relationship and asks whether it holds differently in different slices of the sample. Take mtcars: does a car's weight predict its fuel economy, and does that relationship differ between automatic and manual cars? The habit to break is fitting the model separately inside each subgroup and reading the two outputs side by side.

Weight costs an automatic about 3.8 mpg per 1000 lb and a manual about 9.1, and both slopes are comfortably significant. The reflex is to report that as a real difference, that weight matters more than twice as much for manual cars. The two numbers do look far apart, yet nothing so far has tested whether they are further apart than sampling noise alone would produce. The interaction term runs that test.

The wt:am row is the gap between the two slopes, now estimated as a single number with its own standard error and p-value. Here it is -5.30 with p = 0.001, so the weight penalty really is about 5.3 mpg per 1000 lb steeper for manual cars. The wt coefficient of -3.79 reproduces the automatic-only slope, and adding wt:am recovers the manual slope of -9.08, so the interaction model is simply the two subgroup regressions written as one, with the difference between them made testable. The mechanics of fitting and reading interactions are in Interaction Effects in R. The p-value settles whether the difference is real; it says nothing about whether you went looking for it, and a reviewer worried about fishing wants both questions answered.

What to do about it

You are fine

The wt:am interaction came back at p = 0.001, so the difference between the subgroups is not an artifact of comparing two p-values, it is a difference the data actually support. Report it as the interaction rather than as two separate models, and give the estimate with its confidence interval.

The interval runs from -8.26 to -2.34 and stays clear of zero, which is the same conclusion the p-value gave. If you pre-specified the split, you are finished: state the interaction estimate and its interval and move on. If you did not, you can still report it, but as a secondary and explicitly exploratory finding. A significant interaction with a plausible mechanism behind it is a legitimate result even when it was not planned, as long as you label it as exploratory rather than letting it stand as a confirmatory claim (CONSORT 2010, item 18).

It is fixable

The more common situation is the mirror image, where the subgroups look different and the interaction says they are not. Suppose you had instead claimed that horsepower penalises fuel economy more in automatics than in manuals. Fit it the same way.

Both subgroup slopes are significant, and they are almost identical: -0.0591 and -0.0587. The interaction confirms it, an estimated difference of 0.0004 at p = 0.98, which is indistinguishable from no difference at all, so there is nothing here to report as a subgroup effect. The fix is to drop the split and report the pooled effect, which is what you genuinely have.

One unit of horsepower costs about 0.068 mpg across the whole sample, and pooling estimates that more precisely than either subgroup could on its own. One caveat keeps this honest: interaction tests have low power, so a non-significant interaction is not proof that the effect is constant, only a failure to show that it varies (Matthews and Altman, 1996). Report the pooled effect, and if the subgroups matter to readers, write that the data gave no evidence the effect differed across them.

It is a real problem

Sometimes the entire finding lives inside the subgroup. Picture a trial where the treatment did nothing overall, the difference between arms landing at p = 0.28 across all patients. Split by age, the effect reaches p = 0.03 in patients under 50 and vanishes over 50, and the paper is built around the younger group. Two problems make this indefensible. The split was chosen after the null overall result and was one of several the authors could have tried, so its 0.03 is not the number it appears to be. On top of that the interaction between treatment and age, the only real test of whether age moderates the effect, comes back at p = 0.42, so even at face value the subgroup difference is not there. No wording rescues a result like this, and defending it in the response letter usually leads to a published correction later rather than a stronger paper. The honest path is to report the overall null as the finding, present the age result as a hypothesis for a future study, and state that the analysis was exploratory and unplanned. (The numbers here are illustrative, because built-in datasets rarely hold a clean example of this particular failure.)

How to word your response

Each response restates the concern, says what you did, says what it means for the conclusion, and says where it now appears in the manuscript. Fill in the page placeholders.

You are fine:

The reviewer is right to ask about the status of this subgroup analysis. The comparison of automatic and manual transmissions was pre-specified in our analysis plan, and we now test the difference formally with an interaction term rather than by comparing two separate models. The effect of weight on fuel economy is significantly steeper for manual cars (interaction estimate -5.30 mpg per 1000 lb, 95% CI -8.26 to -2.34, p = 0.001), so the subgroup difference is supported by the data and is not an artifact of running two tests. The interaction estimate and its interval now appear in the Results (Results, page X), and the pre-specification is stated in the Methods (Methods, page X).

It is fixable:

We thank the reviewer for this point. On re-examination our claim that horsepower affects economy differently by transmission does not hold: the interaction between horsepower and transmission is far from significant (estimated difference 0.0004, p = 0.98), so we have no evidence the two slopes differ. We have removed the subgroup contrast and now report the pooled effect (-0.068 mpg per unit of horsepower, p < 0.001), which is estimated with greater precision than either subgroup allowed. The text now states that we found no evidence of effect modification by transmission (Results, page X).

It is a real problem:

The reviewer is correct that the age subgroup was not pre-specified and that we relied on its within-group significance. The overall treatment effect was not significant (p = 0.28), the age split was one of several we could have examined after that result, and the interaction between treatment and age is not significant (p = 0.42), so the data do not support an effect confined to younger patients. We have revised the manuscript to report the overall result as the primary finding and now present the age result only as an exploratory, hypothesis-generating observation that would need confirmation in a study designed to test it (Discussion, page X).

Practice

A reviewer writes:

The authors report that rear axle ratio predicts fuel economy in automatic cars but not in manual cars, and treat this contrast as a real difference between the two groups. The split does not appear to have been planned. Please justify it.

Run this block, then decide which of the three outcomes applies. It prints nothing; inspect the four results yourself.