Stop checking your A/B test every morning

Most testing tools show a live result. It is tempting to look each day and stop the moment it crosses 95%. A paper by Ramesh Johari, Leo Pekelis and David Walsh, “Always Valid Inference”, explains why that habit breaks the statistics.

What the paper reports

The authors say the usual p-values and confidence intervals are “wholly unreliable” when people choose the sample size by continuously monitoring a test. Their own illustration is an A/A test, where both versions are identical. The dashboard’s “chance to beat baseline” still rose above the 95% line at some point while the user watched, which counts as a false positive.

They write that false positive probabilities are “well in excess” of the level a user can tolerate, and that even with 10,000 samples, a size common in online tests, the false positive rate “can easily increase fivefold.” The paper does not give one exact rate for all cases. It depends on how often you look.

The authors’ fix is a different kind of p-value, built into a commercial testing platform. Do not assume your tool does the same. If it does not, the ordinary p-value works as intended only when you choose the end point before you start.

How many visitors that takes

Ron Kohavi and colleagues at Microsoft worked through sample sizes in a KDD paper. They use a formula for 95% confidence and 90% power. Their example is a site where 5% of visitors buy:

• To detect a 5% relative change in conversion rate, you need fewer than 500,000 visitors.
• To detect a 20% relative change, about 30,400.

That is their example, not your store. The point is the shape of the curve: halving the change you want to detect needs about four times the visitors. The same paper recommends running at least a week or two, and then in whole weeks, so that weekday and weekend visitors are both counted.

Many small stores will not reach those numbers on one product page. This is our inference, not the paper’s claim. If that is you, a long test of a large change is more honest than a short test of a small one, and a flat result is a real answer.

A rule to set before you start

Our reading of the two papers, not a quote from them:

• Write down the metric, the smallest change worth acting on, and the end date before the test starts.
• Look at the dashboard to confirm it is running, not to decide.
• Decide on the end date, not on the first significant reading.

What to do this week

1. Take the next test you plan. Estimate your visitors per week to the page being tested and your current conversion rate.
2. Using Kohavi’s formula or any sample-size calculator, check whether the visitors you will get in a few weeks can detect the change you hope for. If not, choose a bigger change or a busier page.
3. Write the end date and the single metric in a note, and send it to yourself so it is timestamped.
4. Do not stop early for a win. If you must stop early for a loss that hurts customers, record that you did.