Do Overlapping Confidence Intervals Mean No Significant Difference?

No. Two 95% confidence intervals can overlap and the difference between the groups can still be statistically significant at p < 0.05. This is one of the most common stats misreadings on dashboards, and it usually points to "let's not roll out the change" when the data actually supports rolling it out. The right answer is to put a CI on the difference and check whether that CI excludes zero.

Why don't overlapping confidence intervals prove non-significance?

"Non-overlapping CIs implies significant" is true. The reverse isn't. Two 95% CIs can overlap by a bit — roughly up to a quarter of the full CI length (about half of one arm) — and the test for the difference can still be significant.

The reason is that the standard error of a difference is not the sum of the two individual standard errors. It's:

SE(diff) = sqrt(SE_A² + SE_B²)

That square-root combination is smaller than SE_A + SE_B. The CI for the difference is therefore narrower than the visual gap between the two individual CIs would suggest.

What is the correct test when two confidence intervals overlap?

Ignore the individual intervals and test the difference directly: run a two-sample test (Welch's t-test is the safe default for means) or, equivalently, build a confidence interval for the difference using SE(diff) = sqrt(SE_A² + SE_B²). If that interval excludes zero, the difference is significant at the matching alpha — regardless of how much the two original CIs overlapped.

Refresher on how a single interval is built: confidence intervals explained; the mechanics of the test itself: the t-test explained.

A Worked Numeric Example

Two groups, equal sample sizes, normal data:

Group A: mean = 100, SE = 2.0  → 95% CI = [96.1, 103.9]
Group B: mean = 105, SE = 2.0  → 95% CI = [101.1, 108.9]

The intervals overlap from 101.1 to 103.9. Many readers conclude "not significant."

Difference test:
  diff = 5.0
  SE(diff) = sqrt(2² + 2²) = 2.83
  95% CI for diff = 5 ± 1.96 × 2.83 = [-0.55, 10.55]   borderline

Check sample sizes — if SE = 1.5 each instead:
  CI_A = [97.1, 102.9],  CI_B = [102.1, 107.9]   (still overlap from 102.1-102.9)
  SE(diff) = sqrt(1.5² + 1.5²) = 2.12
  CI for diff = [0.85, 9.15]   p < 0.05 — significant!

The CIs overlapped, but the difference was significant. The "do the bars touch" eyeball test would have called this a tie.

The 16% Rule (Approximate but Useful)

If both CIs come from samples with similar standard errors, the equivalent of a p ≈ 0.05 difference test is roughly an 83.5% individual CI — not 95%. That's where each CI's half-width is about 1.39 standard errors instead of 1.96.

In practice this means: if you see two 95% CIs whose overlap is less than about 25% of the full length of one interval (about half of one arm), the difference is probably significant at p < 0.05. More than that, probably not. But this is a rule of thumb — it doesn't replace the actual test on the difference.

Rule-of-thumb table (95% CIs, similar standard errors)

What you see	Likely verdict at p < 0.05	What to do
No overlap at all	Significant (guaranteed)	Safe to report a difference
Slight overlap (< ~25% of one CI's length)	Probably still significant	Run the difference test — don't call it a tie
Moderate overlap (25–50%)	Borderline either way	The test on the difference is the only answer
Each mean inside the other's CI	Almost certainly not significant	Don't claim a difference; check power

Interpreting the resulting p-value correctly is its own minefield — see p-values explained.

What to Plot Instead

Skip the side-by-side CIs. Plot the difference directly with its own CI:

Variant A vs Control:  diff = +5.2%, 95% CI = [+1.1%, +9.3%]   ← significant
Variant B vs Control:  diff = +1.8%, 95% CI = [-2.4%, +6.0%]   ← not significant

This is unambiguous: if the CI for the difference excludes zero, the difference is significant at the matching alpha. No overlap-eyeballing required.

When does overlap actually mean "not significant"?

When the overlap is large. If each group's mean falls inside the other group's confidence interval, the difference is essentially never significant at the 95% level — that much overlap corresponds to a test statistic well under the critical value. The misconception only bites in the middle range, where the intervals overlap a little but the difference test would still pass. If you keep landing in that gray zone, the real fix is usually more data — see A/B test sample size.

Code: CI of the Difference in Python

from scipy import stats
import numpy as np

a = np.array([...])  # group A observations
b = np.array([...])  # group B observations

# Welch's t-test (unequal variances) is the safe default
result = stats.ttest_ind(a, b, equal_var=False)

# 95% CI for the difference of means
diff = a.mean() - b.mean()
se = np.sqrt(a.var(ddof=1)/len(a) + b.var(ddof=1)/len(b))
df = result.df  # scipy >= 1.10 returns df
crit = stats.t.ppf(0.975, df)
ci = (diff - crit*se, diff + crit*se)
print(f"diff={diff:.3f}, 95% CI={ci}, p={result.pvalue:.4f}")

Common Pitfalls

• Reporting "no difference" because bars touch: the most common version of this mistake. Always test the difference, not the visuals.
• Using a two-sample test on paired data: for before/after on the same users, use a paired test — it has a much smaller SE and bigger power.
• Multiple comparisons: running 10 difference tests at α=0.05 gives a ~40% chance of at least one false positive. Apply a Bonferroni or FDR correction.
• Equating "not significant" with "no effect": a CI like [-2%, +12%] doesn't show no effect — it shows you don't have enough data to rule one out.