How to set up an A/B test for your newsletters

An A/B test sends two versions of a newsletter and shows which one works better. It replaces guessing with real results.

You will need a newsletter. See How to send or schedule a newsletter.

Two ways to test

You write Variant A and Variant B, with one difference between them, such as the subject line.

• Optimize: both versions go to a small part of your list, for example 10% split equally. After a waiting period, the version with the best open rate or click rate goes to everyone else automatically.
• Compare: both versions go to your whole selected list, split 50/50, with no waiting and no automatic winner. You compare the results yourself.

Before you start

• Choose your goal: more opens, more clicks or more sales.
• Test one thing at a time: the subject line, the sender name, the content or the button. Changing more than one thing makes the results hard to read.
• Use a big enough group to reflect your whole list.
• Decide the test length and how you will pick the winner.

Step 1: write Variant A

Go to Emails, then Newsletters, and click Create. Write your first version with its subject, content and links.

Step 2: create Variant B

Click Create an A/B test, then Confirm. Your newsletter is copied.

Step 3: test settings

Choose Optimize or Compare.

With Optimize, set:

• Test duration: how long both versions run before a winner is picked. 48 to 72 hours usually works. Allow at least 24 hours so parents have time to open.
• Email ratio: the share of your list who receive A and B. With 5%, those contacts get A or B, and the other 95% get the winner. High enough for clear results, small enough to limit the impact of a weaker version.
• Winner: best click rate (engagement with your links) or best open rate (interest in your subject and sender name).

With Compare, there is nothing more to set.

Step 4: contact settings

• Send only to contacts who joined more than X days ago.
• Include these tags: contacts need at least one to receive the newsletter.
• Exclude these tags: contacts with any of these do not receive it.

Step 5: edit Variant B

Scroll down to Variant B and change the one thing you are testing. Click Variant A to go back to the first version. Save your changes.

Step 6: check and launch

Check your list, sender name and address, both subject lines, every link, and how both versions look on desktop and mobile. Then:

• Save and run A/B test: start straight away.
• Save and schedule: pick a date and time.

You can cancel with Cancel A/B test before it starts. Once it runs, leave it alone so the results stay reliable.

Stopping a test

After launch, you cannot change the content or settings. To stop the test, click the three dots and choose Stop A/B test. Anything not sent yet is canceled.

Read your results

Click the three dots, then Stats. Look at your open or click rates, find the winner and the gap between the two. Then ask why: was the subject more inviting, the message clearer, the button easier to spot? Use what you learn in your next newsletter.

Resend to non-openers

Click the three dots next to the finished test, choose Resend to contacts who did not open, set the time and click Duplicate and schedule.