Bayesian vs Frequentist A/B Testing: Which Should You Use?

Frequentist testing asks how surprising your data would be if the variants were identical, and answers with a p-value. Bayesian testing asks how likely each variant is to be better given the data and a prior, and answers with a probability. With weak priors and the same data they usually agree on the numbers; they differ in what the numbers mean and in how easy they are to misuse.

What is the actual difference between the two approaches?

The difference is what gets treated as random. Frequentist methods treat the true conversion rate as a fixed unknown constant and your data as random; Bayesian methods treat the data as fixed once observed and represent your uncertainty about the rate as a probability distribution.

	Frequentist	Bayesian
Parameter is	Fixed unknown constant	Random variable with a distribution
Primary output	p-value, confidence interval	Posterior distribution, P(B > A), credible interval
Requires a prior	No	Yes (can be deliberately uninformative)
Guarantee offered	Long-run error rate (α) across repeated tests	Coherent probability statement given the prior
Decision rule	p < α with a pre-set sample size	P(B > A) above a threshold, or expected loss below a cap
Reports "chance of winning"	No, and cannot	Yes, directly

That last row is the honest reason Bayesian A/B testing spread through product teams: stakeholders want "how likely is B better?" and the frequentist framework structurally cannot answer that question. It answers a different one.

What do the two methods say about the same test?

Almost the same thing, expressed differently. Take control at 500 conversions from 10,000 users and variant at 550 from 10,000, a 10% relative lift.

FREQUENTIST (two-proportion z-test)
  p̂_A = 5.00%   p̂_B = 5.50%   difference = +0.50 pp
  z = 1.585      p = 0.113     → not significant at α = 0.05
  95% CI on the difference: [-0.118 pp, +1.118 pp]

BAYESIAN (Beta-Binomial, uniform Beta(1,1) prior)
  posterior_A = Beta(501, 9501)    posterior_B = Beta(551, 9451)
  P(B > A) = 94.3%
  Expected loss if you pick B = 0.0077 pp
  Expected loss if you pick A = 0.507 pp
  95% credible interval on the difference: [-0.120 pp, +1.119 pp]

The intervals agree to two decimal places, because with a flat prior and this much data the posterior is dominated by the likelihood. What changed is the sentence you can say out loud. Frequentist: "if the variants were truly identical, we would see a gap this big or bigger 11% of the time." Bayesian: "given the data and a flat prior, there is a 94% probability B's true rate exceeds A's." Same evidence, and 94.3% sounds far more decisive than p = 0.113 to a room full of non-statisticians. That asymmetry in perceived decisiveness is the main practical risk of switching.

What is expected loss and why does it matter?

Expected loss is the average amount of conversion rate you give up if you pick a given variant and it turns out to be the worse one. It is the number that makes Bayesian testing genuinely useful for decisions, and it is more informative than P(B > A) alone.

Expected loss(B) = E[ max(rate_A - rate_B, 0) ]

In the test above: 0.0077 percentage points.
On a 5% baseline that is a 0.15% relative downside.

The interpretation is concrete: if you ship B and you are wrong, you expect to lose about 0.008 percentage points of conversion on average. A common rule is to ship when expected loss falls under a threshold you set from business tolerance, such as 0.1% of baseline. That converts the statistics into a caring-about-money statement, and it correctly distinguishes "B is probably better by a hair" from "B is probably better by a lot." P(B > A) = 94% cannot tell those apart; expected loss can.

Does Bayesian A/B testing let me stop whenever I want?

No, and this is the most damaging myth in the field. Continuous monitoring with a "stop when P(B > A) > 95%" rule inflates your error rate just as peeking inflates a frequentist α.

The confusion comes from a true statement: a Bayesian posterior is valid at every point in time given the data seen so far, so there is no "you looked too early" violation of the math. But the decision rule layered on top is a stopping rule, and stopping rules have error rates. If you check daily and stop the first time P(B > A) crosses 95%, then across many genuinely null tests you will hit that threshold far more often than 5% of the time, because you gave yourself many chances. The posterior is honest; the procedure is not.

What actually helps:

• Use expected loss as the stopping rule, not P(B > A). It is far more stable near the start of a test and does not spike on a lucky first day.
• Set a minimum sample size anyway, computed exactly like a frequentist one. See A/B test sample size.
• Run at least one full business cycle regardless of what the numbers say, to absorb day-of-week seasonality and novelty effects.
• If you truly need continuous monitoring, use methods designed for it on either side: alpha spending and group sequential designs, or always-valid confidence sequences.

How do I choose a prior without cheating?

Use a weak, uninformative prior unless you have a defensible reason not to, and always report which prior you used. Beta(1,1) is uniform over [0,1] and is the standard neutral default for conversion rates.

Prior	Meaning	When to use
Beta(1, 1)	Uniform, all rates equally likely	Default; safe, lets the data speak
Beta(0.5, 0.5)	Jeffreys prior, slightly favors extremes	Very small samples, reference analysis
Beta(50, 950)	Centered on 5%, worth ~1,000 prior observations	Strong historical baseline; will visibly pull small-sample results

The rule of thumb: a Beta(a, b) prior is worth roughly a + b pseudo-observations. If your test has 20,000 users per arm, a Beta(50, 950) prior contributes about 5% of the total information and barely moves the answer. If your test has 800 users per arm, that same prior is doing most of the work. Informative priors are legitimate, but they must be chosen before the test and disclosed, or they are just a thumb on the scale.

Which one should my team actually use?

Either works if you fix the sample size in advance and read the interval rather than the headline number. Pick based on your audience and your governance needs, not on which one is philosophically superior.

Situation	Better fit	Why
Regulated or audited decisions	Frequentist	Fixed, universally understood error rates; no prior to defend
Product teams making ship/no-ship calls	Bayesian	Expected loss maps directly to business cost
Many low-traffic tests	Bayesian	Priors and expected loss degrade more gracefully than p-values
Multi-armed tests with a "pick the best" goal	Bayesian	P(best) generalizes past two arms naturally
Comparing against published external results	Frequentist	Everyone else reports p-values and CIs
Team with no statistician on hand	Frequentist	Fewer knobs to set wrong; prior selection is a real failure mode

The genuinely important discipline is identical in both worlds: one pre-registered primary metric, a sample size fixed before launch, a full business cycle of runtime, and a decision based on the size of the effect rather than the significance verdict. Teams that do those four things get similar answers from either framework. Teams that do not get bad answers from both.

Common mistakes in Bayesian and frequentist testing

• Reading a p-value as the probability the null is true. p = 0.113 does not mean an 11.3% chance the variants are identical. It is the probability of data this extreme assuming they are. See p-value explained.
• Reading a 95% confidence interval as a 95% probability that the parameter is in this interval. The frequentist guarantee is about the procedure: 95% of intervals built this way cover the true value. The credible interval is the probability statement people want, which is a real advantage of the Bayesian version.
• Believing Bayesian testing removes the need for sample size planning. It does not. A test with 300 users per arm is uninformative in any framework.
• Shipping on P(B > A) = 92% without looking at magnitude. A 92% chance of a 0.01% lift is not worth the deploy. Check expected loss and the credible interval.
• Switching frameworks after seeing a disappointing result. Running the frequentist test, getting p = 0.11, then rerunning it as Bayesian to report 94% is framework shopping. Pick before launch.
• Using an informative prior fit on the same data you are analyzing. That double-counts the evidence and produces overconfident posteriors.