What P-Values Actually Tell You

A p-value is NOT the probability that your hypothesis is true.

It's the probability of seeing your results (or more extreme) if the null hypothesis were true.

Example

You test if a new website design increases clicks. You get p = 0.03.

Wrong interpretation: "There's a 97% chance the new design is better."

Right interpretation: "If the designs were actually the same, there's a 3% chance I'd see this difference or greater."

Common Threshold

p < 0.05 = "statistically significant" is arbitrary. It means less than 5% chance under the null hypothesis.

What P-Values Don't Tell You

• ❌ The size of the effect (could be tiny but "significant")
• ❌ The importance of the finding
• ❌ Whether it matters in practice
• ❌ Whether you should make a decision