Skip to content
DialogueDialogueDialogue Help Center

Shopify Theme A/B Testing

Thinking about a new Shopify theme, or a big redesign in a theme copy? With a Shopify Themes test you split your shoppers between your current theme and another theme installed in your store, and measure which one performs better before you fully switch.

This test type is available for Shopify stores.

Before you start

Section titled “Before you start”
  • The theme you want to test must already be installed in your Shopify store (in Online Store → Themes). It doesn’t need to be published.
  • Dialogue has to be switched on in every theme in the test. In the theme list, a theme without Dialogue shows an orange warning icon and an Enable Script button. Click it, switch on the Dialogue app embed in that theme, and save. See Set Up A/B Testing on Your Store.

How to set up a theme test

Section titled “How to set up a theme test”
  1. Open A/B testing from the left menu and click Shopify Themes.
  2. Give your test a name, for example “New theme vs current theme”.
  3. For Variant A, click Select Theme on the theme you want to compare against. Your live theme is marked with a star.
  4. For Variant B, click Select Theme on the theme you want to test.
  5. Want to compare more themes? Click Add variant.
  6. Set the Traffic Allocation for each variant.
  7. Open Set Audience and choose the devices the test runs on: desktop, mobile and tablet.
  8. Open Set Test Parameters and choose:
    • Winning Metric: AOV, CVR (conversion rate), Revenue, or Add to cart rate.
    • When you want your test to end?: Once I stop it, Specific date, or Stop test on winner. With Stop test on winner you can also tick Apply winner to all traffic.
  9. Click Start A/B test.

Tip: Just installed or renamed a theme and don’t see it? Click Refresh themes. To change a theme before the test, click Edit in Shopify next to it.

Tips for theme tests

Section titled “Tips for theme tests”
  • Keep everything else the same. Use the same products, prices and apps in both themes, so the theme is the only difference.
  • Run the test for full weeks, so weekdays and weekends count equally.
  • Read How Dialogue Picks an A/B Test Winner to understand when a winner is called.

Next steps

Section titled “Next steps”

Start a Shopify theme test

Need help? Open the chat in your Dialogue dashboard and ask Dia, or ask to talk to a person.