Sample Ratio Mismatch (SRM): How to Detect It With Chi-Square

A sample ratio mismatch (SRM) is when the traffic split your A/B test actually delivered doesn't match the split you configured, for example 50/50 in setup but 49/51 in the logs. It's not a rounding quirk: an SRM means your randomization broke somewhere in the pipeline, and it invalidates every downstream result until you find and fix the cause.

What is a sample ratio mismatch?

It's a statistically significant gap between the traffic allocation you set (say, 50/50 or 90/10) and the allocation your logs actually show. A small gap is expected from random noise; an SRM is a gap too large to be chance, which means something is systematically routing more users into one arm than the other, or dropping them unevenly before they get counted.

How do you test for an SRM?

Use a chi-square goodness-of-fit test comparing observed counts per variant to the expected counts under your configured split. The statistic is chi² = Σ(O-E)²/E, summed across variants, and for a two-arm test it has 1 degree of freedom.

from scipy.stats import chisquare

# 50,000 users, configured 50/50 split
observed = [24500, 25500]
expected = [25000, 25000]

chi2_stat, p_value = chisquare(observed, expected)
print(f"chi2 = {chi2_stat:.2f}, p = {p_value:.6f}")
# chi2 = 20.00, p = 0.000008

Sample size changes how much a given percentage skew matters. The same 5% relative imbalance (95 vs. 105 out of 200 users) gives chi² = 0.5 and p ≈ 0.48 — not significant, because there isn't enough data to distinguish it from noise. At 50,000 users, the same 5% imbalance would be enormous. This is exactly why you run the chi-square test instead of eyeballing the percentages.

Total users	Observed split	Chi²	p-value	SRM?
200	95 / 105	0.50	0.48	No — too small to tell
20,000	9,850 / 10,150	4.50	0.034	Borderline — flags at 0.05, not at 0.001
50,000	24,500 / 25,500	20.00	0.000008	Yes — clear SRM

The middle row is why the threshold choice matters: at p < 0.05 it looks broken, at the stricter p < 0.001 convention it doesn't trigger an alarm yet. Neither answer is wrong; they're different tolerances for false alarms, discussed below.

What causes a sample ratio mismatch?

Almost every SRM traces back to something that treats the two variants differently before or during logging, not to the randomizer itself misbehaving.

• Redirect bias. One variant sends the user through an extra client-side or server-side redirect. Redirects add latency and a failure point, so slower devices, flaky networks, or impatient users disproportionately drop out of that arm before an event ever fires.
• Uneven bot filtering. If bot traffic isn't filtered identically upstream of both variants, a crawler wave can land almost entirely in whichever arm it happens to hash into, skewing the human-only counts you actually care about.
• Tracking or logging failures. A variant that renders slightly slower can miss the analytics beacon on page unload, or a client-side experiment script can throw an error in one variant's code path and never fire the exposure event.
• Uneven bucketing. A caching layer, CDN edge rule, or old cached asset serves a stale variant to part of the traffic; a hashing bug assigns users non-uniformly; or a targeting rule unintentionally overlaps with the experiment's audience filter.

Why does an SRM invalidate the whole test?

Because the mechanism that caused unequal counts almost never affects users at random — it correlates with device, network speed, geography, or bot-ness, and those same factors correlate with your outcome metric. An SRM is not a footnote to report alongside your results; it's a sign the two groups are no longer comparable, so any lift or drop you measured could be entirely an artifact of who got excluded, not what you changed.

Put differently, a valid A/B test depends on random assignment producing two groups that are identical in expectation except for the treatment. An SRM is direct evidence that assumption failed, which means the causal inference the whole test rests on is unsupported — no amount of statistical significance on the primary metric fixes that.

What's a safe SRM threshold to use?

Most experimentation platforms check for SRM automatically on every single test, and that changes the math on false positives. At the standard p < 0.05 cutoff, remember a p-value only tells you how surprising the data would be if the split really were 50/50 — it doesn't say the split is fine, and 1 in 20 perfectly healthy experiments would still get flagged just from noise; run that check across thousands of experiments and you generate a steady stream of false alarms. That's why the common industry convention is a much stricter cutoff for this specific check, typically p < 0.001 (chi² critical value ≈ 10.83 for 1 degree of freedom, versus ≈ 3.84 at p < 0.05).

A stricter threshold trades a slightly higher chance of missing a small, real mismatch for far fewer false alarms on a check you're running constantly — a reasonable trade for an automated gate, less reasonable if you're manually investigating a specific test you already suspect is broken.

Common mistakes with SRM checks

• Eyeballing the split instead of testing it. "52/48, close enough" can be a massive, highly significant SRM at large sample sizes, and a real imbalance at small ones can look scary but be pure noise.
• Trusting results despite a flagged SRM. "But the metric still shows a clear lift" doesn't rescue the test — a biased assignment mechanism can inflate or deflate a metric in either direction.
• Checking SRM only once, at the end. A mismatch introduced by a mid-test deploy can go unnoticed if you only check the final cumulative counts; check daily during the run.
• Stopping at "there's an SRM" without root-causing it. Segment the mismatch by device, browser, and geography before rerunning — otherwise you'll likely reintroduce the same bug.