The Complete Guide To A/B Testing Emails In Sitecore

Email A/B Testing From A To...B

A/B testing your emails is an important aspect of great marketing.

Rather than assuming what your market wants or prefers, it offers you the chance to find out which version they respond better to, which will in turn help you achieve your goals for the marketing campaign.

This guide contains everything you need to know about A/B testing emails in Sitecore.

We’ll explore how to send an email A/B test, as well as how to view the results of your test with both manual and automated options.

Let’s get into it.

How To Send An Email A/B Test

Creating A New Email Campaign

On the Sitecore launchpad, click Email Experience Manager.

 

Click Create.

 

Select Regular Email Campaign

 

Choose the template you want to use, and then click Create.

 

Setting Up A New Campaign

Complete all fields on the General tab:

• Name: Write a descriptive name for the campaign
• Description: Add a description for the email campaign so that anyone who clicks on it will know the details
• Select a campaign group (optional). For more information about campaign groups, read my blog about Sitecore campaigns.
• From name: Choose what ‘From’ name you want your recipients to see (usually your company’s name)
• From email: Choose what email address you want the email sent from
• Reply to email: Choose what email address you want replies to be sent to (usually the same as the ‘from email’). If you leave this blank, it will automatically use the from email as the reply to email

Click on the Recipient tab

Choose the list you want to send emails to with the Include List dropdown.

 

If you want to exclude any recipients from the send, which might already be in the Include List, you can add their email to a suppression list and select it in the Exclude List dropdown (optional).

Click the Message tab.

You’ll see that you’re editing message variant A.

 

Set the email subject, content, and the alternative text version of the email (the version that recipients will see if they can’t view the HTML version.

Creating The B Version

To change only one thing between the A & B version (recommended):

Click the copy symbol.

 

To set a completely different content as the B version: (not recommended).

Add an additional variant (the B version), by clicking the plus sign. NOTE: You can have more variants than just the A/B, but this is not recommended.

 

You’ll now see Variant B appear next to Variant A, and can update the email body, subject and alternative text versions.

 

Removing A Variant

To remove a variant, select the variant you want to remove, and then click the dropdown arrow, and select Remove this variant.

 

To preview how a version will look for recipients, select the version, and then click the eye icon. Choose a recipient from the list and click OK.

 

Selecting Versions And Test Size

Click on the Delivery tab.

Select which versions you want to include in the send. If you’re doing an A/B test, you want to select both of them (or all if you’re doing a multivariate test - not recommended).

Select the ‘Size of the test’ by choosing a percentage from 1-100%. This determines what percentage of recipients will be included in the test. If you choose 10%, only 10% of contacts in the list will enter the A/B test.

The rest of the list will see the winning version (if you choose to select a winner automatically. If you choose to select a winner manually, the A/B test will only send to 10% of contacts in the list, and then you will have to choose the winner manually).

Selecting A Winner

Choose whether you want to select a winner automatically, or manually.

NOTE: If you choose to select a winner automatically, you will not be able to view the final numbers that the winner won by. Sitecore will choose the winner for you, and then send the rest of the emails to the winning version and you will never know how much version A or B was successful by.

If you choose to select the winner automatically: You can choose the amount of time you want the winner to be selected after and whether you want the winner to be chosen based on the open rate (a test of the subject lines), or the value per visit.

If you choose to select the winner manually:You’ll have to return back to this area to choose the winner based on either the open rate, click rate, value or action.

Final Set Up Steps

Choose whether you want to send the email now, schedule the delivery for a specific date or time in the future, or if you want to send it recurrently.

If you want to be notified when message delivery is complete, click the checkbox.

If your email has personalized content, click the Personalization checkbox.

If your email is a service campaign, click the Service email campaign checkbox (so all recipients receive the email regardless of their defined consent settings).

If you have multiple languages configured in your Sitecore instance, you can select the Multi-Language checkbox, which will send the message in the preferred language of each recipient.

To start the A/B test, click Start A/B test, and then click OK.

View The Results Of Your A/B Test Campaign

As mentioned above, the only way to view the results (in numbers) of your campaign, is to choose to select the winner manually.

Click on Email Experience Manager from the Sitecore launchpad.

Click In Progress.

 

Here you’ll be able to see all the A/B tests you have running at the moment, that are waiting for you to choose the winner.

Select the email that you want to choose the winner of, and click the Open campaign icon.

 

Click on the delivery tab

You will see a table that shows the open rate, click rate, value and action for each variant of the email.

Click the Choose as winner button, for the variant that you wish to make the winner.

 

Click OK.

The rest of the contacts in the campaign will be sent the winning version of the emails.

View The Winning Email Sitecore Chose

If you chose to select the winner of the A/B test automatically, Sitecore will send the emails to the size of the test pool you decided on when you chose the % of recipients for the test, and Sitecore chose the winner after the amount of hours that you set, based on the criteria you set the winner to be chosen by.

1. Click on Email Experience Manager from the Sitecore launchpad
2. Click on Sent Email Campaigns
3. Choose the email from the list by selecting the hyperlinked name
4. On the right, you can view a preview of the email that won, or after the preview of the email, click Full Email Campaign to see more details
5. Select Message in the top navigation. The winner will be displayed here

Wrap Up

Sending A/B test emails is easy in Sitecore.

The aspect of A/B test emails that I get asked the most questions about is how to view the results of the A/B test, and whether you should choose the winner manually or automatically.

I hope this blog gave you enough insight into whether you should select the manual or automatic winner option depending on what you’re trying to achieve with the campaign.

If you want to see the exact numbers for each variant of the email in the test, choose to select the winner manually.

If you aren’t interested in seeing the exact numbers for each, and would like Sitecore to choose the winner for you, select the automatic option, but keep in mind that you won’t be able to see the full results of the test, other than knowing which version won.

To find out how to set A/B page tests up, read my blog about Running A/B and Multivariate Tests in Sitecore.

If you have any questions about A/B tests, emails or any and all things marketing and Sitecore, please feel free to ask them in my Facebook group for Sitecore marketers and content authors, or ask me on Twitter @natmankowski and I'll reply to you personally.