How Dialogue Picks an A/B Test Winner
When you create an A/B test, you choose a winning metric: the number that decides which version is better. Dialogue keeps checking the results and tells you when one version is clearly ahead.
Winning metrics you can choose
Section titled “Winning metrics you can choose”| Metric | What it measures | Good for |
|---|---|---|
| Conversion Rate | Share of visitors who placed an order | Most tests |
| Add to cart rate | Share of visitors who added any product to the cart | Product page and collection changes, faster results than orders |
| Revenue | Revenue per visitor | Price, bundle and upsell tests |
| AOV | Average order value | Bundle and free shipping threshold tests |
| Clicks | Clicks per visitor on the tested element | Banners, buttons, navigation |
About Add to cart rate
Section titled “About Add to cart rate”- Shopify stores only. It’s not available on other platforms, so you won’t see it there.
- It counts every add to cart in your store, not only clicks on Dialogue elements. A visitor counts once, even if they add several products.
- It works automatically with the Dialogue app on Shopify. Nothing to install or set up.
- It needs the visitor’s tracking consent, like the rest of your test data.
Add to cart rate usually reaches a result sooner than Conversion Rate, because many more visitors add to cart than buy.
When Dialogue calls a winner
Section titled “When Dialogue calls a winner”To avoid declaring a winner by chance, every version in the test must first reach:
- At least 7 days of running time
- At least 100 visitors
- At least 10 conversions (orders, add to carts or clicks, depending on your metric)
After that, a version is the winner only when Dialogue is at least 95% sure it beats the original. With more than one variant, the bar is raised for each one, so testing many versions at once doesn’t produce lucky winners.
Until then, the results page shows the numbers so far with “Not enough data yet” or “No clear difference”.
Automatic actions
Section titled “Automatic actions”If your test is set to stop or move traffic to the winner automatically:
- It waits at least 14 days, even if a winner shows up earlier. Results often swing in the first days.
- Only Conversion Rate and Add to cart rate winners are applied automatically.
- Revenue and AOV winners are shown on the results page but never applied automatically. Review them and apply the winner yourself.
- Run tests for full weeks (7, 14, 21 days), so weekdays and weekends are counted equally.
- Don’t stop a test as soon as one version pulls ahead. Early leads often disappear.
- Low traffic? Test two versions, not three or four. Each extra version needs more visitors.