Simpson's Paradox

Simpson's Paradox is when a trend that appears in every subgroup reverses or disappears once the groups are combined. It happens because a lurking variable is distributed unevenly across groups, so the aggregate mixes apples and oranges.

How Can a Trend Reverse When You Aggregate Data?

Aggregation weights each subgroup by its size, so if one group is concentrated in a "hard" segment and the other in an "easy" one, the overall rates reflect the mix, not the underlying performance.

Treatment A beats B in small stones AND large stones,
yet B wins overall — because A got mostly hard cases:

              A               B
Small     81/87  (93%)    234/270 (87%)
Large    192/263 (73%)     55/80  (69%)
Overall  273/350 (78%)    289/350 (83%)  ← reversal

What Is the Berkeley Admissions Example?

In 1973, UC Berkeley's aggregate graduate admissions looked biased against women (44% of men admitted vs. 35% of women), yet within most departments women were admitted at equal or higher rates.

Dept   Men: applied/admitted   Women: applied/admitted
A          825 → 62%                108 → 82%
B          560 → 63%                 25 → 68%
C          325 → 37%                593 → 34%
E          191 → 28%                393 → 24%
F          373 →  6%                341 →  7%

Aggregate: Men 44%  vs  Women 35%

Women applied disproportionately to competitive departments (C, E, F) with single-digit-to-30% admit rates, while men flooded departments A and B where most applicants got in. The department was the lurking variable; the aggregate reversed the department-level picture.

How Do You Guard Against Simpson's Paradox?

Segment by every plausible confounder before drawing a conclusion from an aggregate rate, and check whether the group mix differs between the populations you're comparing.

import pandas as pd

# Always look at both levels before concluding:
df.groupby("variant")["converted"].mean()             # aggregate
df.groupby(["segment", "variant"])["converted"].mean() # per segment

# Red flag: the segment mix differs by variant
pd.crosstab(df["segment"], df["variant"], normalize="columns")

If the per-segment story and the aggregate story disagree, report the segmented numbers and explain the mix shift. Which level is "true" depends on the causal question — this is a close cousin of the correlation vs. causation problem.

Common Pitfalls

• Comparing rates across populations with different mixes: year-over-year conversion can "drop" purely because cheaper traffic grew, even if every channel improved.
• Segmenting after seeing the result: decide your confounders up front; slicing until the story flips is p-hacking in reverse.
• Assuming the segmented view is always right: if the "segment" is a consequence of the treatment (a collider or mediator), conditioning on it introduces bias instead of removing it.
• Trusting dashboards that only show totals: aggregate KPIs hide mix shifts by construction.