The Multiple Comparisons Problem: How to Stop Inflating False Positives

Every test you run at α = 0.05 has a 5% chance of a false positive when nothing is going on. Run 10 tests and the chance that at least one comes back "significant" by luck is 40%, not 5%. That is the multiple comparisons problem, and it is why a dashboard with 20 metrics almost always shows a winner somewhere.

Why does testing more metrics inflate false positives?

Because each test gets its own independent 5% chance to fire, and those chances compound. The probability of at least one false positive across m independent null tests is called the family-wise error rate (FWER).

FWER = 1 - (1 - α)^m        with α = 0.05

Tests (m)	Family-wise error rate	Bonferroni threshold (α/m)
1	5.0%	0.05
2	9.8%	0.025
5	22.6%	0.010
10	40.1%	0.005
20	64.2%	0.0025
50	92.3%	0.001

This is why "we tested the new checkout and it did not move conversion, but it lifted mobile session length for logged-in users in Germany" is not a finding. It is the arithmetic above doing its job. If your metrics are correlated, the real FWER sits somewhere below these numbers, but it is still far above 5%.

How does the Bonferroni correction work?

Divide your α by the number of tests and compare every raw p-value to that stricter threshold. Testing 10 metrics at an overall 5% error rate means each individual p-value must be below 0.005.

Adjusted threshold: α* = α / m           (compare raw p to α*)
Equivalent form:    p_adjusted = min(1, p × m)   (compare to α)

10 metrics, α = 0.05  →  each p must be < 0.005

Bonferroni controls the FWER: the probability of even one false positive across the whole family stays at or below 5%. It makes no assumption about independence, which is why it is safe on correlated metrics. The cost is power. At m = 20 you need p < 0.0025, and real effects that would comfortably clear 0.05 vanish.

Use Bonferroni when a single false positive is genuinely expensive: a drug safety endpoint, a pricing change, a claim you will put in a press release. Use it when m is small, ideally under 10.

How does Benjamini-Hochberg differ from Bonferroni?

Benjamini-Hochberg (BH) controls the false discovery rate (FDR) instead of the family-wise error rate. It accepts that some proportion of your declared discoveries will be wrong, and caps that proportion, which recovers a lot of the power Bonferroni gives away.

Benjamini-Hochberg procedure (FDR level q = 0.05):
1. Sort the m p-values ascending: p₍₁₎ ≤ p₍₂₎ ≤ … ≤ p₍m₎
2. For each rank i, compute the critical value (i / m) × q
3. Find the LARGEST i where p₍ᵢ₎ ≤ (i / m) × q
4. Reject the null for ranks 1 through i (all of them, even any
   whose own p-value exceeded its own critical value)

Here is the same set of 10 p-values run through no correction, Bonferroni, and BH at q = 0.05:

Rank i	p-value	BH critical (i/10 × 0.05)	Bonferroni (0.005)	BH
1	0.001	0.005	reject	reject
2	0.004	0.010	reject	reject
3	0.011	0.015	—	reject
4	0.014	0.020	—	reject
5	0.021	0.025	—	reject
6	0.041	0.030	—	—
7	0.240	0.035	—	—
8	0.390	0.040	—	—
9	0.550	0.045	—	—
10	0.870	0.050	—	—

Uncorrected, six results look significant. Bonferroni keeps two. BH keeps five: rank 5 is the largest rank whose p-value (0.021) falls under its critical value (0.025), so ranks 1 through 5 are all rejected. The interpretation of the BH set is "roughly 5% of these five discoveries are expected to be false," not "each of these five is individually safe at 5%."

Which correction should I use?

Match the correction to the cost of being wrong: Bonferroni when one false positive is unacceptable, BH when you are screening many candidates and can tolerate a known fraction of duds.

	Bonferroni	Benjamini-Hochberg
Controls	Family-wise error rate	False discovery rate
Guarantee	P(≥1 false positive) ≤ α	Expected share of false ones among discoveries ≤ q
Power	Low, drops fast as m grows	Much higher, degrades gently
Assumptions	None (holds under any dependence)	Independence or positive dependence
Good for	m < 10, high-stakes single claims	m in the dozens to thousands, screening
Typical use	Confirmatory A/B guardrails	Feature selection, exploratory metric scans

A practical middle path for experiments: designate one primary metric up front and test it at the full α = 0.05, then treat every other metric as exploratory and either correct it or label it as a hypothesis for a follow-up test. That is how most mature experimentation programs handle it, and it is covered in the planning step of our hypothesis testing walkthrough.

Is peeking at results a multiple comparisons problem?

Yes. Every time you check an in-flight test and are willing to stop on p < 0.05, you have run another test. Checking daily for two weeks is closer to 14 comparisons than one, and the true false positive rate climbs well past 5%.

Sequential peeking is not exactly the independent-test case in the table above, because the looks are strongly correlated with each other, but the direction is the same and the inflation is severe. The standard fixes:

• Fix the sample size in advance and do not look at significance until you reach it. Simplest and most reliable. See statistical power for how to set it.
• Use a sequential testing method built for it: alpha spending (O'Brien-Fleming, Pocock boundaries), group sequential designs, or always-valid p-values / confidence sequences. These are designed to be monitored continuously.
• If you must peek, peek for harm only. Watching a guardrail metric to abort a disaster is fine; the asymmetry is that you never stop early to declare a win.

Common mistakes with multiple comparisons

• Counting only the tests you reported. The correction applies to every comparison you looked at, including the segments you sliced and discarded. If you checked 14 segments and reported the one that moved, m = 14.
• Reading a corrected p-value as an ordinary one. A Bonferroni-adjusted p of 0.04 at m = 10 means the raw p was 0.004. Do not mix adjusted and raw values in the same table without labeling them.
• Applying BH but rejecting only the p-values that individually beat their critical value. Step 4 rejects everything at or below the largest passing rank, including ranks that failed their own threshold. Skipping that step makes BH more conservative than it should be.
• Correcting for exploratory work and calling the survivors confirmed. A corrected exploratory finding is still exploratory. Confirm it in a fresh, pre-registered test.
• Forgetting that a single "significant" segment can just be Simpson's paradox or small-sample noise. Check the segment's sample size before you build a roadmap on it.
• Assuming a non-significant corrected result proves no effect. It does not; correction lowers power, so read the confidence interval, not just the verdict.