For Shopify and online store owners

What A/B tests, AOV and AI‑search studies actually found.

Check your store against what the research found, before you spend on a test. 2 to 3 minute notes, every source linked.

1in3ideas tested at Microsoft succeeded, on average.Peer-reviewedKohavi et al., KDD

Sources include Kohavi et al., W3C, Google and Apple.

1in3ideas tested at Microsoft succeeded, on average.

Peer-reviewedKohavi et al., KDD (opens the source)Read the note: Most A/B tests lose: what large testing programs report

How the money moves, in $ and %

How average order value, conversion and margin go up

Revenue is three numbers multiplied together, and profit depends on the margin you keep. Each app works on one of them. Each card shows what research on large retailers found for that lever, not what any store will get.

Average order value

More items or more units in each order

More units shoppers intended to buy when a price promotion was stated for multiple units, Manning and Sprott found

Peer-reviewedManning and Sprott, Journal of Retailing (opens the source)

Work it out in $ with your numbersDoes a bundle pay for itself in your store?

The finding, and how the apps work on it

The effect “is contingent on the magnitude of the quantity specified in the offer and the rate of product consumption”, and it measured intentions, not orders. Drechsler and colleagues found “X for $Y” did better than “X + N free”, and that quantity requirements of four to five units showed the largest effects, in functional categories but not hedonic ones. BulkPilot’s tiers are 2 and 3 units; quantity 4 and above continues at the Buy 3 rate.

When a shopper adds the pot and the card to the plant, or takes three instead of one, the order is bigger and one pick-and-ship is spread over more revenue. In a McKinsey retail case, shoppers judged a two-item bundle on its one total price.

  • BundlePilot: shoppers build the bundle themselves and see one total.
  • BulkPilot: Buy 2 and Buy 3 prices next to Add to cart.

Peer-reviewedDrechsler et al., European Journal of Marketing (opens the source) Consulting firmMcKinsey, “Dynamic pricing in retail” (opens the source)

Conversion rate

Fewer reasons to leave before paying

12% of US online shoppers who abandoned a cart, excluding those just browsing, couldn’t see or work out the total cost up front

Independent researchBaymard, “Cart Abandonment Rate Statistics” (opens the source)

Work it out in $ with your numbersDo Buy 2 and Buy 3 tiers pay off?

The finding, and how the apps work on it

That is one reason on Baymard’s list. Overall, 70.22% of online carts are abandoned, on average across 50 studies, and most abandonment has causes no app fixes, such as shoppers who were only browsing. Baymard also estimates that $260 billion of lost US and EU orders are recoverable through better checkout design. That covers every checkout fix it has documented, so it is context for this lever, not a figure for any one app.

Two of the reasons shoppers leave are ones these apps work on: not seeing the total until checkout, and leaving checkout to hunt for a code.

  • BulkPilot: the quantity price shows before checkout and applies automatically, with no code.
  • BundlePilot: the bundle total and the saving show on the product page, in the cart and on the order.

Independent researchBaymard, “Payment UX” (opens the source)

Margin you keep

Discounts at the planned depth, on the planned hours

400–800 basis points of margin from full markdown optimisation, not scheduling alone

Consulting firmMcKinsey, “Hitting the mark” (opens the source)

Work it out in $ with your numbersWhat does a sale that runs late cost?

The finding, and how the apps work on it

McKinsey found that getting the depth, timing and choice of discounts right can improve margin rates by 400 to 800 basis points.

Price falls straight to profit: McKinsey calculated that a 5% price cut needs 18.7% more volume just to break even.

  • SalePilot: the end time is enforced and every price goes back, with a report of what reverted.

Consulting firmMcKinsey, “The power of pricing” (opens the source)

Popular advice, next to its source

Four tips you meet in almost every guide, and what the research behind them found.

  1. Popular claim: Red buttons convert better.

    What the source found: Nonepublished data behind it that we could check

    Contrast with what surrounds a button has the better-documented basis.

    Elliot; Laws of UX (opens the source)Read the note: Colour psychology: what the research supports and what it doesn’t

  2. Popular claim: Add a decoy option and shoppers pick the bigger pack.

    What the source found: 0 of 27studies with a product people could taste, touch or see found the effect

    A 38-study paper found the effect in number tables and mostly nowhere else. Where people could see or touch the product, it found no significant effect.

    Frederick, Lee and Baskin, JMR (opens the source)Read the note: The decoy price: why the famous trick is shakier than it sounds

  3. Popular claim: Stop the test once it says 95% to win.

    What the source found: 5×how far false positives can rise, even at 10,000 samples

    Watching a test and stopping at the first significant reading makes the result unreliable. Even at 10,000 samples, false positives can rise fivefold.

    Johari, Pekelis and Walsh (opens the source)Read the note: Stop checking your A/B test every morning

  4. Popular claim: Repeat your keywords so AI answers pick you.

    What the source found: 17.8for keyword stuffing, against 19.5 for the unchanged page

    In the GEO benchmark, keyword stuffing scored below the unchanged page: 17.8 against 19.5. Citing sources and plain, fluent text did better.

    Aggarwal et al., KDD (opens the source)Read the note: What the research says about getting cited in AI answers

How each note is checked

  • Each note ends with What we checked: every source it uses, linked so you can open it yourself.

  • Where we read only a paper’s abstract and not the full text, its source line says so.

  • Checks we ran ourselves are listed as Our check. Where we ran none, the note says so and makes no first-hand claim.

The main sources behind the four notes up top

From the end of a note: The decoy price: why the famous trick is shakier than it sounds

What we checked

  1. Source Frederick, Lee and Baskin, “The Limits of Attraction”, Journal of Marketing Research, vol. 51, no. 4, pp. 487 to 507 · full text Open on marketing.wharton.upenn.edu (opens the source)
  2. Source Huber, Payne and Puto, “Let’s be honest about the attraction effect”, Journal of Marketing Research, vol. 51, no. 4, pp. 520 to 525 · abstract only Open on scholars.duke.edu (opens the source)
  3. Our check We have not run a decoy test on any store, so this note makes no first-hand claim about results.