What Is Statistical Power and Why Is 80% the Standard?

Statistical power is the probability that your test detects a real effect of a given size. It is 1 - β, where β is the false negative rate. An underpowered test does not just fail to reach significance; it burns weeks of traffic and then produces an inconclusive result you cannot distinguish from "no effect," which is the most expensive outcome an experiment can have.

What does statistical power actually mean?

Power is the probability that you reject the null hypothesis, given that a specific alternative hypothesis is true. It is always power to detect an effect of a stated size, never a property of the test on its own.

That last point trips people up. "This test has 80% power" is incomplete. The full statement is "this test has 80% power to detect a 10% relative lift on a 5% baseline at α = 0.05." The same test has far less power against a 3% relative lift and far more against a 30% one. Power depends on four inputs, and fixing any three determines the fourth:

• Significance level (α) — usually 0.05. Loosening it raises power and raises false positives.
• Effect size — the smallest difference you actually care about, often called the minimum detectable effect (MDE). Bigger effects are easier to detect.
• Sample size (n) — the only lever you fully control.
• Variance — noisier metrics need more data.

How do I calculate sample size for a given power?

For a two-proportion test comparing conversion rates, the per-arm sample size is a direct formula. Plug in your baseline rate, the rate you want to be able to detect, and the z-values for your α and power.

n per arm = (z_(α/2) + z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²

  z_(α/2) = 1.960   for α = 0.05, two-sided
  z_β     = 0.842   for 80% power
  z_β     = 1.282   for 90% power

Example: p₁ = 0.050, p₂ = 0.055 (a 10% relative lift)
  (z_(α/2) + z_β)² = (2.8016)²          = 7.84888
  p₁(1-p₁) + p₂(1-p₂) = 0.0475 + 0.051975 = 0.099475
  (p₁ - p₂)²       = (0.005)²           = 0.000025

  n per arm = 7.84888 × 0.099475 / 0.000025 = 31,230.7 → 31,231

Note how brutally the denominator punishes small effects: it is squared. Halving the lift you want to detect multiplies the required sample by four. Our A/B test sample size guide walks through the same arithmetic with a calculator-style checklist.

What happens if my test is underpowered?

You get a coin flip dressed up as an experiment. Below are the power levels for the 5.0% → 5.5% test at various sample sizes, all at α = 0.05 two-sided.

Users per arm	Power to detect a 10% relative lift	Chance you miss a real effect
5,000	20%	80%
10,000	35%	65%
20,000	61%	39%
31,231	80%	20%
50,000	94%	6%

There is a second, nastier consequence. Among the underpowered tests that do hit significance, the observed effect is systematically inflated, because only unusually large sample estimates cleared the threshold. This is the "winner's curse" of experimentation: the low-powered wins you ship rarely replicate at the size you measured.

Why is 80% power the convention?

It is a convention, not a law. Jacob Cohen proposed β = 0.20 in the 1960s on the reasoning that a false positive is about four times as costly as a false negative in academic publishing, giving the 4:1 ratio of β = 0.20 to α = 0.05.

Whether that ratio fits your situation is a business question, not a statistical one:

• Use 90–95% power when the change is cheap to ship and missing a real win is the expensive error, or when you plan to make a permanent decision from one test.
• 80% is fine for routine, reversible experiments where you can retest.
• Below 80% is rarely worth running. If the traffic math says 45% power, you are not running an experiment; you are buying a lottery ticket with four weeks of traffic.

Going from 80% to 90% power on the example above raises the requirement from 31,231 to 41,809 per arm, roughly 34% more traffic. That is the actual price of the extra certainty.

What if I do not have enough traffic?

Increase the effect size you are willing to test for, reduce the variance of the metric, or stop running that test. Those are the honest options; padding the runtime after seeing the data is not one of them.

• Raise the MDE. Accept that you can only detect a 20% relative lift, and only test changes big enough to plausibly produce one. At 5% → 6% you need just 8,155 per arm.
• Pick a less noisy metric. Moving from revenue per user (heavy-tailed) to conversion rate (binary) often cuts required sample dramatically. Trimming or winsorizing extreme values helps too; see outlier detection.
• Use variance reduction. CUPED and similar pre-experiment covariate adjustments routinely cut variance 20–50% on metrics with strong pre-period correlation.
• Test bigger changes. A button color test on a low-traffic site is unanswerable. A redesigned checkout flow might not be.

Common mistakes with power

• Computing power after the fact from the observed effect. "Post hoc power" calculated from the result you got is a deterministic function of the p-value and tells you nothing new. Power is a design-time calculation using the effect size you care about, not the one you happened to see.
• Treating a non-significant result as proof of no effect. With 35% power, a null result is close to uninformative. Report the confidence interval on the difference; if it spans -8% to +12%, you have simply not learned anything.
• Powering for a lift you would never actually see. Assuming a 30% relative lift makes the sample requirement pleasantly small and the test useless, because the realistic 3% effect will never clear the bar.
• Peeking at results daily and stopping at the first p < 0.05. This inflates the real false positive rate well past 5%. See the multiple comparisons problem for what repeated looks cost you.
• Splitting power across many metrics. Sizing for your primary metric does not power the twelve secondary metrics on the same dashboard. They are all underpowered by construction.