Flight log / Transmission 030

Landing page testing without lying to yourself

Most landing page tests don't fail. They get misread. At Google and Bing, only about 10% to 20% of experiments produce positive results (Kohavi and Thomke, HBR, 2017), so a team that finds a winner every other week should be asking harder questions.

The math of A/B testing is settled. The behavior around it isn't. Most bad calls come from people, not formulas: a test stopped on a good Tuesday, a "win" found in the fourteenth segment someone checked, a variant that looked great for a week because it was new.

Here's how to stop fooling yourself. No tool talk, just the habits that make any testing setup worth the traffic you feed it.

What is landing page testing?

Landing page testing is a controlled experiment. You randomly assign visitors to the control (the page you run today) or a variant, hold everything else equal, and compare one primary metric. For paid traffic that's usually a form completion, a demo request or a qualified lead, not time on page.

The randomization is the whole trick. It's what lets you say the page caused the difference, rather than the week, the campaign mix or the weather. Anything that breaks randomization, like sending one ad group to A and another to B, turns your experiment into a comparison of two audiences.

Three formats cover almost everything you'll run:

• A/B test: one control, one variant. The default.
• A/B/n test: several variants against one control. Each extra arm needs its own share of traffic.
• Multivariate test: several elements changed in combination. Statistically elegant, traffic hungry, rarely the right call for B2B paid traffic.

How do you write a landing page test hypothesis?

A hypothesis is a prediction that can be wrong. "Let's try a new hero" isn't one. A useful format has five parts: the evidence, the change, the audience, the metric and the minimum effect you'd act on.

Because [evidence], we believe [change] for [audience] will move [primary metric] by at least [minimum effect]. We'll call it after [sample size per variant].

Notice the guardrail in the second example. A shorter form that doubles junk leads isn't a win. Pick one primary metric, then name the one or two guardrail metrics that must not get worse.

Should you test one variable or take bigger swings?

The textbook says change one thing so you know what caused the result. The textbook assumes you have traffic. Small changes produce small effects, and small effects need big samples to detect.

The pattern that shows up in most B2B accounts: not enough paid traffic per page to detect a 5% lift on anything. That makes bold variants the practical default. Test a different offer, a different angle or a different page structure. Once a direction wins, use follow-up tests to find out which part did the work.

Small changes can matter at scale. Kohavi and Thomke describe a small change to how Bing displayed ad headlines that lifted revenue 12%, worth more than $100 million a year in the US alone. But Bing could detect that because it had enormous traffic. Your 600-visit-a-week demo page can't.

How much traffic does a landing page test need?

Sample size depends on three inputs: your baseline conversion rate, the smallest lift worth detecting, and how much error you'll tolerate. Here's a hypothetical page converting at 4%, using the standard two-proportion calculation at 95% confidence and 80% power:

Halve the effect you want to detect and you need roughly four times the traffic. That single fact explains most abandoned test roadmaps. For turning visitors into calendar days, and why tests should cover full weeks, see how long to run an A/B test.

Why is peeking at test results a problem?

Classic significance tests assume you check once, at a sample size you chose in advance. Every extra look is another chance for random noise to cross the line. Stop at the first crossing and your false positive rate climbs fast.

Evan Miller's well-known worked example shows how fast: a 50% baseline, a variant with no real effect, a significance check after every visitor, and a stop at the first 5% significant result or 150 observations, whichever comes first. That setup wrongly declares a winner 26.1% of the time. That's more than five times the error rate the dashboard implied.

Three honest ways out:

1. Fixed horizon. Decide the sample size up front, look only for broken tracking, and read the result once.
2. Sequential designs. Pre-plan a small number of interim looks with stricter thresholds at each one.
3. Bayesian decision rules. Report the probability a variant is better and the expected cost of being wrong. Useful, but not a free pass to peek. More in Bayesian vs. frequentist A/B testing.

What is the novelty effect in A/B testing?

A novelty effect is a lift that comes from something being new, not better. A 2012 KDD paper from Kohavi and colleagues at Microsoft describes users who "click everywhere" on a new design, creating a bias that "dies quickly if the feature is not truly useful." The mirror image is the primacy effect: returning users perform worse on a changed page until they relearn it, which favors the control.

The same paper suggests three checks: plot the difference between control and variant over time and look for a trend, compute the result for new users only, or exclude the first week, since the gap "usually stabilizes after a week." The authors also add some humility: in their experience, suspected novelty and primacy effects often turned out to be statistical artifacts rather than real behavior. Check before you blame the shiny new page.

For paid landing pages, most visitors are first-timers, which limits novelty bias. Retargeting traffic is the exception. If returning visitors make up a real share of the test, split them out and check whether the lift holds for new visitors alone.

How do segment results mislead you?

The test is flat overall. Then someone slices by device, channel, region, company size and day of week, and there it is: the variant won on mobile LinkedIn traffic from mid-market companies. Ship it?

Not yet. At a 5% significance level, each comparison has a 1 in 20 chance of a false positive when nothing is going on. Check 20 independent segments and the chance at least one "wins" by luck is about 64% (1 minus 0.95 to the 20th power). Check 10 and it's about 40%. That's arithmetic, not bad luck.

• Name segments before launch. One or two pre-registered segments are fair game. Everything found afterward is a lead, not a result.
• Re-test segment wins. Run a fresh test targeted at that segment. If it was real, it shows up again.
• Watch the traffic mix. If a bid strategy change mid-test shifts the share of traffic from one channel, the blended conversion rate can move even when neither page changed. Keep campaign settings frozen while a test runs.

What should you test first on a landing page?

Start where the visitor's decision is made, not where the design team's opinions are strongest. For paid traffic, that order is usually:

Item one is often a build task more than a test. If every ad group lands on the same generic page, start by sending each ad group to the most relevant page you have. The message match guide covers how to audit that gap.

How should you log landing page test results?

A testing program is only as smart as its memory. Logging is the wax on, wax off part of the job: dull, repetitive, and the reason your fifth test is better than your first. Record every test, including the losers and the flat ones.

Kohavi and Thomke point to Twyman's law as a working rule: any figure that looks interesting or different is usually wrong. A surprising 40% lift deserves a tracking check and a replication before it deserves a slide.

Where this breaks

• Very low traffic. Some B2B pages will never reach a valid sample for anything but huge effects. There, a structured qualitative review and sending traffic to better-matched pages beats a test that never finishes.
• Conversions that aren't revenue. A form completion win can be a pipeline loss. Where volume allows, check downstream qualification rates before calling it.
• Moving platforms. Ad platforms adjust delivery constantly. Freezing settings helps, but your traffic mix is never perfectly still.
• Small wins don't stack neatly. Five 10% wins rarely add up to 61% in production. Effects interact, and some wins were partly noise.

FAQ