Effect Size vs Statistical Significance: Which One Should You Act On?

Statistical significance tells you whether an effect is distinguishable from zero. Effect size tells you whether it is big enough to care about. With a large enough sample, a completely trivial difference will be significant, so a p-value on its own is not a decision. The number you act on is the effect size and its confidence interval.

What is the difference between effect size and significance?

Significance is a statement about evidence against the null hypothesis; effect size is a statement about magnitude. They answer different questions and can disagree in both directions.

	Statistical significance	Effect size
Question answered	Is the difference distinguishable from zero?	How large is the difference?
Depends on sample size	Yes, heavily	No
Typical measures	p-value, z, t	Cohen's d, relative lift, absolute difference, r, odds ratio
Units	Unitless probability	Metric units or standard deviations
Drives the decision	Screens out noise	Yes, this is the business input

The trap is that only one of these grows with your traffic. Collect enough data and any nonzero difference, however tiny, eventually crosses p < 0.05.

Why does a big sample make trivial effects significant?

Because the p-value depends on the effect size divided by the standard error, and the standard error shrinks with the square root of n. The effect stays the same; the yardstick gets finer.

Here is one fixed, real difference (5.00% vs 5.10% conversion, a 0.10 percentage point absolute gap and a 2% relative lift) tested at four sample sizes, α = 0.05 two-sided:

Users per arm	z	p-value	Verdict	Effect size
50,000	0.72	0.470	not significant	+0.10 pp (2% rel.)
100,000	1.02	0.307	not significant	+0.10 pp (2% rel.)
200,000	1.44	0.149	not significant	+0.10 pp (2% rel.)
500,000	2.28	0.022	significant	+0.10 pp (2% rel.)

The last row is the one people misread. Nothing about the change got better between rows three and four. You just bought enough traffic to resolve a difference that was never worth shipping engineering time for. Sample size is a measurement instrument, not evidence of importance.

How do I calculate Cohen's d?

Cohen's d expresses the gap between two group means in pooled standard deviations, so it is comparable across metrics with different units. Divide the difference in means by the pooled standard deviation.

d = (M₂ - M₁) / s_pooled

s_pooled = sqrt( ((n₁-1)s₁² + (n₂-1)s₂²) / (n₁ + n₂ - 2) )

Example: M₁ = 100, M₂ = 102, s_pooled = 15
  d = (102 - 100) / 15 = 0.133   ← a small effect

Conventional benchmarks (Cohen, 1988):
  0.2 small | 0.5 medium | 0.8 large

Now hold d fixed at 0.133 and vary n, using a two-sample t-test:

n per group	Cohen's d	t	p-value
100	0.133	0.94	0.347
1,000	0.133	2.98	0.003
10,000	0.133	9.43	< 0.0001

Same effect, three different verdicts. The d column is the one that stayed honest. Treat Cohen's benchmarks as rough conventions rather than rules; in a mature conversion funnel, d = 0.05 can be worth millions, and in a psychology lab d = 0.2 can be noise.

What effect size should I report for a conversion test?

For conversion rates, report the absolute difference in percentage points, the relative lift, and a confidence interval on the difference. Cohen's d is awkward for proportions and rarely how the business thinks.

Control:   5.00%    Variant: 5.10%     n = 500,000 per arm

Absolute difference: +0.10 percentage points
Relative lift:       +0.10 / 5.00 = +2.0%
95% CI on the difference: [+0.014 pp, +0.186 pp]
                     i.e. [+0.28%, +3.72%] relative

That interval is the whole answer. It excludes zero (so it is significant), and it tells you the plausible range runs from "basically nothing" to "3.7% better." If your minimum worthwhile lift was 5%, the interval rules that out entirely: this is a significant result that also constitutes evidence against the effect being big enough to matter. A p-value alone cannot express that. The same logic drives our note on overlapping confidence intervals: put the interval on the difference, not on each group.

Can a non-significant result still have a meaningful effect size?

Yes, and this is the mirror-image error. An underpowered test can show a large observed effect with a p-value of 0.30 simply because the interval is too wide to exclude zero.

If a test shows a +12% relative lift with a 95% CI of [-4%, +29%], the correct summary is "inconclusive, and consistent with anything from a small loss to a large win," not "no effect." Concluding "no difference" from a wide interval is the classic misuse of a null result. The remedy is more data, not a stronger adjective. See statistical power for sizing, and Type I vs Type II error for why false negatives are so easy to overlook.

Common mistakes reading effect size and significance

• Treating p < 0.05 as a shipping decision. Significance is a filter for noise, not a measure of value. Pair it with a pre-agreed minimum worthwhile effect.
• Reading the p-value as the probability the null hypothesis is true. It is not. A p-value is the probability of data at least this extreme assuming the null is true. See what a p-value actually means.
• Reporting relative lift without the baseline. "+40%" on a 0.05% baseline is +0.02 pp. Always show both.
• Trusting the observed effect from a barely-significant, low-powered test. Effects that squeak past the threshold in small samples are systematically inflated, so the replication is almost always smaller.
• Comparing Cohen's d across metrics with different variance structures as if the benchmarks were universal. The 0.2/0.5/0.8 labels are conventions from behavioral research, not physical constants.
• Dropping the confidence interval from the summary slide. The interval carries both pieces of information at once; the p-value carries half of one.