What Is a t-Test?

A t-test checks whether the difference between means is real or just sampling noise. It comes in three flavors: one-sample (a mean vs. a fixed value), two-sample (two independent groups), and paired (the same subjects measured twice).

When Should You Use Each Type of t-Test?

Use a one-sample t-test to compare a group mean against a known target, a two-sample t-test to compare two separate groups, and a paired t-test when each observation in one group has a natural partner in the other.

One-sample: "Is our average delivery time really 30 minutes?"
Two-sample: "Do users on variant B spend more than users on A?"
Paired:     "Did the same users spend more after the redesign?"

The paired test is more powerful for before/after data because it removes person-to-person variation. Running a two-sample test on paired data throws that power away.

How Do You Run a t-Test in Python with scipy?

scipy.stats has one function per flavor: ttest_1samp, ttest_ind, and ttest_rel. Each returns a t-statistic and a p-value.

from scipy import stats
import numpy as np

rng = np.random.default_rng(42)
a = rng.normal(30, 5, 50)   # group A
b = rng.normal(32, 5, 50)   # group B

# One-sample: is the mean of a equal to 30?
stats.ttest_1samp(a, popmean=30)
# TtestResult(statistic=0.839..., pvalue=0.405...)

# Two-sample (independent groups):
stats.ttest_ind(a, b, equal_var=False)  # Welch's t-test
# TtestResult(statistic=-0.763..., pvalue=0.448...)

# Paired (same subjects, before/after):
stats.ttest_rel(a, b)

Pass equal_var=False for the two-sample test. That runs Welch's t-test, which stays valid when the groups have different variances and costs almost nothing when they don't.

What Are the Assumptions of a t-Test?

The t-test assumes observations are independent, the data (or paired differences) are roughly normal, and for the classic two-sample version, the groups have equal variance.

1. Independence  - each observation drawn separately
2. Normality     - matters most for small n; with n > 30
                   the CLT makes the test robust to skew
3. Equal variance - only for classic ttest_ind;
                    use equal_var=False to drop it

The p-value it produces tells you how surprising your data would be if the means were truly equal. See what a p-value actually means before you act on one.

Common Pitfalls

• Using a two-sample test on paired data: before/after measurements on the same users need ttest_rel, not ttest_ind.
• Ignoring independence: multiple sessions from the same user are not independent observations. Aggregate to one row per user first.
• Tiny samples with heavy skew: with n < 15 and skewed data, consider the Mann-Whitney U test instead.
• Testing after peeking: a t-test is one step in a workflow — state the hypothesis and alpha first. See the full hypothesis testing steps.