Most A/B tests lose: what large testing programs report

One well-documented source on this is a paper by Ron Kohavi, Alex Deng, Roger Longbotham and Ya Xu, “Seven Rules of Thumb for Web Site Experimenters” (KDD). The authors drew it from thousands of controlled experiments at Amazon, Booking.com, LinkedIn and several Microsoft sites. Here is what they report.

Most ideas fail. At Bing, they write, most experiments fail, and the ones that succeed improve key metrics by 0.1% to 1.0% once diluted to overall impact. They put Bing’s success rate on ideas at about 10 to 20 percent, and the average across several Microsoft experiments at about one in three.

Big wins happen, but rarely. Kohavi and Stefan Thomke tell one such story in Harvard Business Review. A Bing employee proposed a change to how ad headlines were displayed. It sat as a low priority for more than six months. When it was finally tested, it raised revenue by 12% and became the best revenue idea in Bing’s history. The paper treats wins like this as the exception: small changes can have a big impact, but changes rarely do.

A result that looks too good is usually a bug. The paper names Twyman’s law: “Any figure that looks interesting or different is usually wrong.” When a test shows a surprising jump, the authors investigate before they celebrate, and most of the time they find a bug.

Someone else’s win may not be yours. The authors say they are skeptical of many amazing A/B results reported in the literature, and that an idea which worked on one site may not work as well on another. Their advice is to replicate.

Speed showed up again and again. In a Bing experiment they slowed some users by 100 and 250 milliseconds. Every 100 milliseconds of speedup was worth 0.6% of revenue.

What this means for a store

These numbers come from sites with very large traffic. In our reading, three things carry over to a store of any size:

• Expect most tests to lose or to show nothing. Plan more than one, and decide in advance what you will do if the test is flat.
• Treat a huge lift as a reason to check your tracking, not as a win. Run it again before you change the store for good.
• Don’t copy a published win. Test it on your own traffic, because, as the authors put it, an idea that worked on one site may not work as well on another.

A store with modest traffic also cannot detect changes as small as the 0.1% to 1.0% the paper describes. That is our inference, not the paper’s claim.