Flight log / Transmission 033
How long should an A/B test run? The honest answer
"Until it looks good" is not a duration. At a 2% conversion rate, spotting a 20% lift with standard settings takes about 21,100 visitors per variant, and no amount of refreshing the dashboard makes that number smaller.
> Run an A/B test until it reaches the sample size you calculated before launch, and for at least one full week, ideally two full business cycles. Sample size comes from four inputs: baseline conversion rate, minimum detectable effect, power and significance level. Low traffic means testing bigger changes or waiting longer, not calling it early.
Most tests that "won" and then quietly stopped winning were never winners. They were stopped on a good Tuesday. The fix is boring: decide how long the test runs before it starts, then leave it alone.
Below: the math, the calendar rules stacked on top of it, when you're actually allowed to stop, and what to do when your B2B landing page gets 400 visitors a week. If you want the wider program around this, start with our landing page testing guide.
// 01 · the briefingWhat decides how long an A/B test should run?
Two clocks, and the test has to beat both. First, a sample size: enough visitors in each variant to detect the effect you care about. Second, a calendar minimum: enough time to cover the normal swings in who visits and when. Divide the total sample by your weekly traffic, round up to whole weeks, and compare it to the calendar minimum. The longer of the two is your duration.
The sample size itself comes from four inputs:
| Input | What it means | Effect on duration |
|---|---|---|
| Baseline conversion rate | How often the current page converts today | Lower baseline, longer test |
| Minimum detectable effect (MDE) | The smallest lift worth detecting, usually stated as relative change | Smaller MDE, much longer test |
| Statistical power | The chance of detecting the MDE if it is real. 80% is the common default | Higher power, longer test |
| Significance level | The false positive rate you accept. 5% (95% confidence) is the common default | Stricter level, longer test |
The MDE does the most damage. Required sample size scales roughly with one over the square of the effect, so halving the lift you want to detect roughly quadruples the visitors you need. Evan Miller's widely cited rule of thumb, n = 16σ²/δ², shows the same relationship: the effect (δ) sits squared in the denominator.
Pick the MDE from the business, not from hope. If a 5% lift wouldn't change a single decision, don't pay the traffic bill to detect one.
How many visitors does an A/B test need?
Here is a worked, hypothetical table. Say a 20-person SaaS team sends 2,000 paid visitors a week to one landing page and splits them 50/50.
Assumptions: two variants, equal split, two-sided test, 95% significance (α = 0.05), 80% power, conversion measured per visitor. Sample sizes use the standard normal-approximation formula for comparing two proportions: n per variant = (1.96 × √(2p̄(1 − p̄)) + 0.84 × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)², where p₁ is the baseline, p₂ the target rate and p̄ their average. The z values are shown rounded; the table uses the exact values (1.95996 and 0.84162). Weeks are rounded up to whole weeks.
| Baseline | Relative MDE | Target rate | Visitors per variant | Total visitors | Weeks at 2,000/week |
|---|---|---|---|---|---|
| 2% | 10% | 2.2% | 80,682 | 161,364 | 81 |
| 2% | 20% | 2.4% | 21,109 | 42,218 | 22 |
| 2% | 30% | 2.6% | 9,798 | 19,596 | 10 |
| 3% | 50% | 4.5% | 2,518 | 5,036 | 3 |
| 5% | 10% | 5.5% | 31,234 | 62,468 | 32 |
| 5% | 20% | 6.0% | 8,158 | 16,316 | 9 |
| 5% | 30% | 6.5% | 3,780 | 7,560 | 4 |
| 10% | 20% | 12.0% | 3,841 | 7,682 | 4 |
| 10% | 30% | 13.0% | 1,774 | 3,548 | 2 |
Hypothetical example, computed for this post. Online calculators use slightly different formulas (pooled vs. unpooled variance, continuity corrections, one or two sided), so expect results within a modest margin of these, not identical numbers.
Read the 2% rows again. Detecting a 10% relative lift on a 2% page would take this team more than a year and a half. That isn't a test. That's a hobby. The same team can detect a 30% lift on a 10% page in two weeks.
Two practical notes. First, the baseline must match the metric you will judge on. If the page converts at 5% to form fill but you plan to call the test on demo bookings at 1%, size the test on 1%. Second, more variants means more traffic. Each added variant needs its own full sample, and comparing several variants against control without a correction raises your false positive rate.
// 03 · orbitWhy should a test cover full business cycles?
Because your Monday morning visitor and your Saturday night visitor are different people. In most B2B accounts the pattern jumps out once you look: weekday traffic is heavier and more qualified, weekends are thinner and noisier. A test that runs Tuesday to Friday compares two variants on an audience you don't actually have.
The rules that survive contact with real traffic:
- Run whole weeks. One full week is the floor. Two full weeks is a better floor, because one unusual week (a holiday, a conference, a competitor's launch) can still skew a single cycle.
- Match your real cycle. If buyers typically return two or three times over a few weeks before converting, the test should span that window, or late converters land in your results after you've already called it.
- Account for conversion lag. Google Ads Help recommends running Google Ads experiments for at least 4 to 6 weeks, or longer if you have a long conversion delay. The same logic applies to page tests judged on downstream outcomes.
- Don't run forever. Very long tests pick up their own problems: cookies clear, people switch devices and land in the other variant, campaigns change underneath you. Once you're past two months on one test, the honest move is usually to redesign it with a bigger MDE.
Freeze the campaign while the test runs. New ads, new bids or a new audience mid-test means you're measuring two experiments and learning from neither.
When is it OK to stop an A/B test?
A standard fixed-sample test assumes you look at the result once, at the end. Every extra look is another chance to catch random noise at a lucky moment. Evan Miller's essay on the problem shows how fast this compounds: if you peek 10 times and want a real 5% false positive rate, each look has to clear a 1% significance threshold. Most dashboards don't adjust for that. They show 95% confidence on day three and let you believe it.
Write these stopping rules down before launch:
- Stop when the planned sample and the calendar minimum are both met. Then read the result once.
- Stop early only for harm. Set a guardrail in advance, such as a broken form or a sharp drop in a critical metric, and kill the test if it trips. That's a safety call, not a winner call.
- Stop early for a win only with a method built for it. Sequential testing designs adjust thresholds for repeated looks. If you want to peek, choose one of those up front. Our Bayesian vs. frequentist comparison covers the tradeoffs.
- Call "inconclusive" when it's inconclusive. No significant difference at the planned sample means the effect, if any, is smaller than your MDE. That's a result. Log it and move on.
- Check the split. If a 50/50 test ends at 54/46, something in the setup is broken (redirects, bot traffic, tracking). Fix it before trusting any number.
Unless you own a time machine with a flux capacitor, you can't go back and un-peek. The only protection is deciding before the first visitor arrives.
// 05 · payloadWhat can low-traffic B2B teams do instead?
Most B2B landing pages don't get thousands of visitors a week. That doesn't make testing pointless. It changes what you test and how you read it.
| Option | How it helps | The catch |
|---|---|---|
| Test bigger swings | A new offer, a new page structure or a different headline promise can plausibly move conversion 30% or more, which cuts sample size sharply | You learn that the package won, not which piece did it |
| Pool similar traffic | Run the same change across several ad groups or pages with similar intent so one test gets combined volume | Only valid if the change means the same thing on every page |
| Use a higher-volume metric | A primary metric like form start or CTA click converts more often than demo booked, so it needs fewer visitors | Proxies can move without the outcome moving. Keep the real outcome as a guardrail |
| Accept more risk, out loud | Lower power or a 90% confidence level shortens the test | More missed winners and more false ones. Document the setting with the result |
| Use a sequential or Bayesian design | Lets you check results and stop when evidence is strong enough | It doesn't create traffic. Small samples still give wide uncertainty |
| Don't split test at all | Customer interviews, session recordings and sales call notes find obvious problems fast | Directional, not proof. Validate the big calls when volume allows |
The pattern that shows up in most B2B accounts: teams test button colors on 300 visitors a week, wait six weeks, get nothing and conclude testing doesn't work. The traffic was fine for a bolder question. Fewer, bigger tests beat many tiny ones when volume is scarce. If you're tempted to shift traffic toward a leader as you go, read up on multi-armed bandit testing first, since that answers a different question than a classic A/B test.
Low traffic isn't a reason to skip testing. It's a reason to stop asking small questions.
How do you set an A/B test duration before launch?
- Pick the primary metric you will judge on, and find its baseline from at least the last 4 weeks of data.
- Set the MDE from the business case: the smallest lift that would change a decision.
- Choose significance and power. 95% and 80% are sensible defaults. Write down any deviation and why.
- Calculate sample size per variant, multiply by the number of variants, and divide by weekly eligible traffic.
- Round up to whole weeks, apply a two-week floor, and extend for your conversion lag.
- If the answer is longer than about two months, raise the MDE by testing a bolder change, or pool traffic.
- Record the end date, the stopping rules and the guardrails before the test goes live.
Where this breaks
The math above is honest. It also leans on four assumptions that real traffic likes to break:
- Independent visitors. The formula assumes each visitor is one independent trial. Repeat visitors, shared company networks and cross-device journeys bend that, usually making real uncertainty larger than the formula says.
- A stable baseline. Paid traffic changes when budgets, bids or seasonality change. A baseline from a quiet month can understate how long a busy month needs, and the reverse.
- Novelty effects. A new design can win early just for being new and fade later. Longer tests catch some of this. Nothing catches all of it.
- Lead quality. A variant can lift form fills and lower qualified pipeline. If you can, check downstream quality before rolling out a winner.
Something's warming up on the launch pad.
We're building something for people who read posts like this. Early access opens in waves.
Early access, by invitation.
FAQ
What is the minimum time to run an A/B test?
At least one full week so every weekday is represented, and ideally two full business cycles. The test also has to reach the sample size you calculated before launch. Whichever takes longer wins.
How many visitors do you need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you want to detect. At 95% significance and 80% power, a 5% baseline needs about 8,200 visitors per variant to detect a 20% relative lift, while a 2% baseline needs about 21,100 per variant.
Can you stop an A/B test as soon as it reaches significance?
Not with a standard fixed-sample test. Checking repeatedly and stopping at the first significant result inflates false positives. Either commit to a sample size in advance or use a method designed for early stopping, such as a sequential test.
How long should a Google Ads experiment run?
Google Ads Help recommends running experiments for at least 4 to 6 weeks, or longer if you have a long conversion delay.
What if my B2B site doesn't have enough traffic to A/B test?
Test bigger changes so the effect you need to detect is larger, pool traffic from similar pages into one test, use a higher-volume primary metric with care, or accept lower confidence and say so. Some questions are better answered with customer research than a split test.