A/B testing on Shopify: when it’s worth it, how much traffic you need and what to test first

Nico Bértolo · 2026-10-18

In short

• An A/B test only works with enough purchases behind it. At a 2% conversion rate, detecting a 20% lift needs about 21,000 visits per variant.
• Below that volume, don’t test buttons: fix what any audit flags and compare comparable periods before and after.
• Test big changes on the pages that sell most (product page, cart), not a button colour on the homepage.

An A/B test shows two versions of a page to two random groups and keeps the one that sells more. It works, as long as there’s enough traffic for the result to mean something.

How much traffic you really need

• Under 10,000 visits a month: skip classic A/B tests. Fix the obvious and measure before and after.
• 10,000 to 50,000: test only big changes on high-traffic pages.
• Over 50,000: run a continuous testing calendar, one or two tests at a time.

What to test first

Shipping cost and delivery time next to the price, product-in-use photos, reviews next to the buy button and a free-shipping bar in the cart. Changes to the information people use to decide move far more than aesthetic tweaks.

How to run a test on Shopify

Use a dedicated testing app that splits traffic and reports revenue per visitor. Decide the metric and duration upfront, never stop a test because it’s “winning”, and analyse mobile and desktop separately.

Not enough traffic to test? We audit your store, fix what holds sales back and measure conversion before and after. Guaranteed by contract. Book a call

A low-traffic test doesn’t tell you what works. It tells you what happened by chance that week.

Nico Bértolo, 224

Frequently asked questions

How much traffic do I need for an A/B test on Shopify?

At a 2% conversion rate and looking for a 20% lift, about 21,000 visits per variant (around 420 purchases each) at 95% confidence.

How long should an A/B test run?

At least two full weeks and until the planned sample size is reached.