Regression to the Mean: Why Extreme Groups Improve Anyway

Regression to the mean is the tendency of an extreme measurement to be followed by a less extreme one, purely from statistical noise, with no real change in the thing you're measuring. If you select the worst-performing group on a noisy metric and measure them again later, they will look better on average even if you did nothing at all, because part of why they looked so bad the first time was random chance that isn't guaranteed to repeat.

What is regression to the mean?

It's a statistical consequence of measuring anything imperfectly. Any observed score is a mix of a true underlying value plus noise, and when you select people specifically because their observed score was extreme, you're disproportionately selecting people who also got unlucky (or lucky) noise on that measurement. Remeasure them, and the noise component isn't correlated across measurements, so it tends to land closer to zero the second time, pulling the average back toward the population mean.

Why do the worst performers improve on their own?

Because "worst performers" is a selection made on a single noisy snapshot, and that snapshot is a combination of true skill and bad luck. Some of the people in that bottom group are consistently weak, but some are average performers who happened to have an off week. On the next measurement, the consistently weak stay weak, but the unlucky-average performers bounce back toward their real level, pulling the group average up even with no intervention.

The size of the effect depends on how noisy the metric is, captured by the correlation r between the first and second measurement (also called test-retest reliability). Lower r means more noise relative to true signal, which means more regression:

expected_retest = mean + r * (observed - mean)

With mean = 50 and a bottom group observed average of 25, here's how the predicted retest average changes with reliability:

Test-retest correlation (r)	Predicted retest average	Apparent "gain"
0.9 (very reliable metric)	27.5	+2.5
0.6 (moderately noisy)	35.0	+10.0
0.3 (very noisy metric)	42.5	+17.5

A noisier metric produces a bigger "improvement" from regression alone, which is the opposite of what most people intuitively expect from a random artifact.

How much will an extreme group regress?

Use the formula directly: multiply the group's distance below (or above) the mean by (1 − r) to get the expected regression, or multiply by r to get how much of the original gap is expected to persist. At r = 0.6, a group starting 25 points below the mean is expected to close 40% of that gap (10 points) by pure regression, landing at 35, while 60% of the original gap (15 points) is expected to persist as their real, stable difference from average.

This is exactly why "we coached the worst performers and they improved by 10 points" is weak evidence on its own. If regression alone predicts a 10-point gain at r = 0.6, an observed 10-point gain in the coached group is fully consistent with the coaching having done nothing.

How do I guard against regression to the mean?

Use a control group drawn from the same extreme selection. Split the bottom performers randomly into a treated group and an untreated control group before doing anything else. Both groups start with the same observed average and the same expected regression, so both will drift back toward the mean on their own. The difference between the two groups' retest averages is your estimate of the true treatment effect, with the shared regression canceling out.

# Bottom-decile group, split randomly before treatment
treated_retest_avg = 36.0     # coached group, second measurement
control_retest_avg = 34.5     # untreated group, same selection, second measurement

true_effect = treated_retest_avg - control_retest_avg
print(true_effect)   # 1.5 — the real effect, with shared regression removed

Without the control group, you'd have looked at the treated group's 25 → 36 change and claimed an 11-point win. With it, the honest estimate is 1.5 points — most of the apparent gain was regression that would have happened anyway.

Common mistakes with regression to the mean

• No control group in a before/after comparison. "We launched X and the metric improved" says nothing about regression unless you compare against a group that didn't get X.
• Selecting on a noisy single-period metric, then reacting to the next period. The noisier the metric (the lower the test-retest r), the larger the artifact — and the more convincing the "improvement" looks.
• Confusing this with the novelty effect. Novelty is a real behavioral reaction that fades over time; regression to the mean is a statistical artifact of remeasuring an extreme group. Different mechanisms, different fixes, and they can occur together.
• Treating it as a data quality problem. It isn't a bug to be cleaned up — it's a mathematical consequence of imperfect measurement plus selecting on extremes, and it will happen with perfectly clean data.