Type I vs Type II Error: What Is the Difference?

A Type I error is a false positive: you declare an effect that is not there. A Type II error is a false negative: a real effect exists and you miss it. Their rates are α and β, you control both through your design, and lowering one while holding sample size fixed always raises the other.

What are Type I and Type II errors?

They are the two ways a hypothesis test can be wrong, defined by the combination of what is true in reality and what your test concluded. The 2x2 table is the fastest way to hold them straight.

	H₀ is true (no real effect)	H₀ is false (real effect exists)
Test rejects H₀ ("significant")	Type I error — false positive probability = α	Correct detection probability = 1 - β (power)
Test fails to reject H₀ ("not significant")	Correct non-detection probability = 1 - α	Type II error — false negative probability = β

The standard mnemonic: a Type I error is crying wolf when there is no wolf, a Type II error is missing the wolf that is actually there. In a product context, Type I means shipping a change that does nothing (or worse, one you now believe in), and Type II means killing a change that would have worked.

Why can't I just make alpha very small?

Because α and β sit on opposite ends of the same threshold. Moving the significance cutoff further out to reject fewer true nulls also makes it harder to reject false ones, so false negatives rise.

Here is a single test design, 25,000 users per arm, looking for a 5.0% → 5.5% conversion lift, evaluated at three α levels:

α (Type I rate)	Critical z	Power (1 - β)	β (Type II rate)
0.10	1.645	81%	19%
0.05	1.960	71%	29%
0.01	2.576	47%	53%

Nothing about the world changed across those rows; only the threshold did. Tightening α from 0.05 to 0.01 cut the false positive rate by a factor of five and nearly doubled the false negative rate. The only lever that improves both columns at once is more data, which is exactly what the power calculation is for.

Which error is worse?

It depends entirely on the cost of each mistake in your situation, and the 4:1 default (α = 0.05, β = 0.20) is a convention, not an answer.

• Type I is worse when acting is expensive or hard to reverse. A drug approval, a pricing change, an architectural rewrite, a public claim. Here you want a stricter α, and you accept losing some real effects.
• Type II is worse when the change is cheap and reversible and the opportunity cost is high. A copy change on a landing page, a low-risk UI tweak. Missing ten real wins to avoid one false positive is a bad trade.
• Type II errors are systematically underreported. A false positive eventually shows up as a change that did not deliver. A false negative shows up as nothing at all: the idea was killed, no one revisits it, and the loss never appears in a review deck.

The practical version of this is to price both errors before designing the test. If a false positive costs roughly $50k in engineering rework and a false negative costs $200k in forgone revenue, your α should be looser than your β, not the other way around.

How do base rates change the picture?

α is not the probability that a significant result is wrong. That depends on how many of the hypotheses you test are true in the first place, and in most experimentation programs that share is low.

Suppose you run 1,000 experiments, only 10% of your ideas actually work, α = 0.05, and power = 80%:

Truly effective ideas:   100  →  80 detected (power)      20 missed  (Type II)
Truly null ideas:        900  →  45 "significant" (α)     855 correctly null

Significant results total:  80 + 45 = 125
Share of those that are false positives: 45 / 125 = 36%

So more than a third of your "wins" are noise, even though α is a well-behaved 5%. Drop power to 50% and it gets worse:

Power	True positives	False positives	Share of wins that are false
80%	80	45	36%
50%	50	45	47%
35%	35	45	56%

This is why underpowered testing programs are worse than they look: low power does not just cost you missed wins, it poisons the wins you do declare. It is also why a p-value is not the probability the null is true: p = 0.05 says nothing about the base rate, and the base rate is doing most of the work above.

How does this relate to precision and recall?

It is the same 2x2 table with different labels, which is why the machine learning framing often clarifies the statistics one. A Type I error is a false positive, a Type II error is a false negative.

Statistics term	ML term	Definition
Type I error rate (α)	False positive rate	FP / (FP + TN)
Type II error rate (β)	Miss rate (1 - recall)	FN / (TP + FN)
Power (1 - β)	Recall / sensitivity	TP / (TP + FN)
1 - α	Specificity	TN / (TN + FP)
—	Precision	TP / (TP + FP)

Precision is the row that has no clean classical-statistics equivalent, and it is the one the base rate calculation above was computing (125 significant results, 80 correct, precision = 64%). If you already think in confusion matrix terms, the mapping is exact, and ROC curves are just the α/β tradeoff drawn as a curve across every possible threshold.

Common mistakes with Type I and Type II errors

• Reading α as "5% of my significant results are wrong." α is conditional on the null being true. The share of your wins that are false depends on the base rate and power, and is usually much higher than 5%.
• Treating "not significant" as "no effect." That is accepting the null, which a hypothesis test never licenses. It means you did not gather enough evidence, which is a statement about your sample, not the world.
• Optimizing α while ignoring β entirely. Most teams have a firm 0.05 rule and no stated power target at all, which fixes one error rate and lets the other float wherever traffic happens to land.
• Peeking, which quietly inflates the real α. Checking daily and stopping on the first p < 0.05 pushes the actual Type I rate well above the nominal one. See the multiple comparisons problem.
• Calling a Type III error a Type I. Getting the direction of a real effect backwards, or answering the wrong question correctly, is a separate failure that no α adjustment fixes.