Flight log / Transmission 031

Bayesian vs. frequentist A/B testing for marketers

Same test, same data, two verdicts. In the hypothetical below, 40 vs. 52 conversions on 1,000 visitors each is "not significant" to one method and "about 90% likely better" to the other, and both are telling the truth.

Statisticians have been arguing about this for about a century, and they're not close to done. You don't need to pick a side in that fight. You need to know what the number on your testing dashboard actually means, and which reading fits the call you have to make on Thursday.

No calculus below. One worked example, small enough to check on a napkin.

What's the difference between Bayesian and frequentist testing?

The two schools disagree about what "probability" means.

• Frequentist: probability is a long-run frequency. The true conversion rate is fixed; your data is the random part. The method imagines rerunning your test thousands of times and asks how often you'd see a result this extreme by chance.
• Bayesian: probability is a degree of belief. You start with a prior (what you believed before), add data, and get a posterior (what you believe now). The conversion rate itself gets a probability distribution.

Hence the different-looking dashboards. A frequentist result is a statement about your data, assuming no difference. A Bayesian result is a statement about the variants themselves.

What does a p-value actually tell you?

The American Statistical Association's 2016 statement defines a p-value, informally, as the probability under a specified statistical model that a summary of the data would be equal to or more extreme than what you observed.

In A/B terms: if the two pages truly convert the same, how often would random noise produce a gap at least this big? A p-value of 0.03 means about 3% of the time.

Now the part dashboards skip. The same ASA statement says p-values do not measure the probability that the studied hypothesis is true, and do not measure the size or importance of an effect. So "p = 0.05" is not "95% chance B is better." It's the most common misreading in testing, and most dashboards practically invite it.

Frequentist results usually ship with a confidence interval. A 95% interval comes from a procedure that would capture the true value in 95% of repeated experiments. That's a claim about the method's track record, not a promise about the one interval on your screen.

What does a Bayesian A/B test tell you?

A Bayesian readout gives you three numbers that sound like the questions marketers actually ask:

1. Probability to beat control. "There's an 88% chance B converts better than A."
2. Credible interval. "There's a 95% probability the true lift is between X and Y." This one does mean what it sounds like, given the model and prior.
3. Expected loss. If you pick B and you're wrong, how much conversion rate do you give up on average?

Expected loss is the underrated one. Chris Stucchio's widely cited 2014 decision rule sets a "threshold of caring": if the expected loss of choosing a variant falls below a difference you wouldn't care about, you choose it. That turns statistics into a business decision with a stated tolerance for error.

The catch is the prior. With a neutral (flat) prior and a decent amount of data, it barely moves the answer. With a strong prior, it can move it a lot. And a prior chosen after you've seen the data is just grading your own homework.

Bayesian vs. frequentist A/B testing: side by side

A worked example with simple numbers

Hypothetical: say a small SaaS team splits traffic to a demo page evenly. After 1,000 visitors each:

The observed lift is 30% relative, 1.2 percentage points absolute. Now the two readings.

The frequentist reading

A standard two-proportion z-test (pooled, no continuity correction) gives a two-sided p-value of about 0.20. That's well above the usual 0.05 bar, so the result is "not statistically significant." The 95% confidence interval for the difference runs from about -0.6 to +3.0 percentage points. It includes zero.

Translation: if the pages were identical, a gap at least this big would still show up about 1 time in 5. You can't rule out "no difference."

The Bayesian reading

With a flat prior (Beta(1,1), meaning no opinion going in), the posterior says, rounded:

• Probability B beats A: about 90%.
• 95% credible interval for the difference: about -0.6 to +3.1 percentage points. Nearly identical to the confidence interval.
• Expected loss if you choose B: about 0.05 percentage points. Expected loss if you stay with A: about 1.2 points.

Translation: B is probably better, the size of the lift is very uncertain, and if B turns out worse, you'd lose very little on average.

So which verdict is right?

Both. Same data, and with a flat prior the intervals are nearly twins. What differs is the question. The frequentist reading protects you from declaring a winner on noise. The Bayesian reading tells you how to bet if you have to decide now.

If B is a headline swap you can reverse in five minutes, a 90% chance of being better with tiny expected loss is a reasonable bet. If B means rebuilding the page template, get more data. For context, a classic fixed-sample design built to detect a 20% relative lift from a 4% baseline (two-sided, 95% confidence, 80% power) needs about 10,300 visitors per variant, roughly ten times this sample. See how long to run an A/B test.

Can you peek at a Bayesian A/B test?

You'll hear that Bayesian tests are immune to peeking. Half true. The posterior is a valid summary of your beliefs whenever you look. But stop the moment it crosses a threshold and you still raise the odds of shipping something that doesn't help.

David Robinson simulated this in 2015: for a change that actually hurt performance, stopping early on a Bayesian rule led to switching to the worse variant in 11.8% of cases, versus 2.5% without early stopping. For comparison, Evan Miller's frequentist simulation, a deliberate worst case that checks significance after every observation up to 150, pushed a nominal 5% false positive rate to 26.1%.

The fix is identical in both camps: write the stopping rule before launch, then obey it. More on the human side of this in landing page testing without lying to yourself.

When should marketers use Bayesian or frequentist testing?

The pattern that shows up in most B2B accounts is the second row: modest traffic, decisions that can't wait six months, and changes that are easy to roll back. That's where Bayesian framing earns its keep, as long as nobody treats 90% as a guarantee.

How should you report A/B test results to stakeholders?

Whichever method you run, report the same four things, so nobody has to do live statistics translation in the meeting:

1. The observed difference, absolute and relative. "B converted at 5.2% vs. 4.0%, a 1.2 point (30%) lift."
2. The uncertainty range. "The true difference is plausibly anywhere from slightly negative to about 3 points."
3. The decision rule you set before launch. "We said we'd ship at 95% confidence" or "we said we'd ship when expected loss fell below 0.1 points."
4. The decision and its reversibility. "Shipping B, monitoring weekly, easy to roll back."

Leading with the range instead of a verdict does more for your testing program's credibility than any choice of method. People forgive a test that came back uncertain. They don't forgive a "winner" that quietly evaporates next quarter.

Where this breaks

• Neither fixes bad data. Broken tracking, uneven traffic splits, novelty effects and bot traffic poison both methods equally.
• Low traffic is low information. Bayesian output always gives you a number. With little data, that number comes with a very wide interval. Read the interval, not just the headline probability.
• Priors can be gamed. Use a neutral or documented prior, set before the test.
• Many variants inflate "probability to be best." With five arms, one will look best by luck. Hold the bar higher as arms increase.

FAQ