Running A/B and Multivariate Tests in Sitecore

Introduction

This blog is part 2 of my ‘Setting Up Sitecore Personalization’ series. If you haven’t read part one yet, click here to read it and follow along first.

Once you’ve finalized your engagement value scale, set up your goals and become familiar with tracking contacts and website visitors in Experience Profile, you can start to configure personalization for your website. Let's start with the basics.

---

Personalization Tactics

Every time a visitor comes to your website they perform a series of activities like browsing pages, submitting a form, downloading content or making a purchase. You can use this activity data to personalize their experience on your site and you can think of personalization simply as a series of if/then rules.

There are 2 main types of personalization in Sitecore:

• Explicit
• Implicit

Explicit personalization is based on rules, i.e. completing a form, favouriting an item, making a commerce transaction, reaching a marketing score.

Implicit personalization is based on pattern matches, i.e. how a user engages with your site, what pages they browse and in what order they browse them.

You can personalize based on virtually anything in Sitecore, all the way from information about their acquisition (i.e. the channel, campaign or referring site a prospect has come from), to their activity (i.e. what pages they’ve viewed, what goals they’ve reached or what keywords they’ve searched for), to information about the prospect’s visit (what country they’re browsing from, what device they’re using, how many times they’ve visited the site, or what time and date they’re visiting). The list is virtually endless.

Before we go any further, when you personalize content you’re gathering visitor information so it’s important that you comply with the European Union's General Data Protection Regulation (GDPR) in Sitecore. You can find out more information about Sitecore and GDPR here.

It’s also important to mention that when it comes to personalization, you don’t want to personalize every component on a page. You want to be subtle about it to nudge them in the right direction, because visitors can find it frustrating when everything on a site starts changing and updating based on their browsing, especially if they’re just clicking around to look at everything.

---

Personalization Tests

When you implement personalization, you want to start off by using tests. Tests do just what they imply - they allow you to see whether you’re giving visitors the best possible experience on your site by tracking whether a variant of a page, component, combination of components or content produces better results. Tests are the most basic way to implement adaptive personalization.

Types of tests:

• Basic content test
• Component test
• Page test

It goes without saying, but always remember that the content or components that you’re testing should aim to achieve the same goal to ensure a fair and accurate test and results, so you can optimize your lead generation.

---

Basic Content Tests

A basic content test compares the old version of a page to the new version of a page.

1. In the Sitecore launchpad, select Content Editor.
2. Select the page that you want to run the basic content test on in the content tree.
3. Select Versions in the top navigation ribbon. Then Add. This will make a new version of the page.
4. Make the changes that you would like to test against the original version of the page. You can do this in Experience Editor or in the Content Editor. Click Save.
5. In the Content Editor select the page that you just made a new version of in the content tree.
6. In the top navigation select Review, then Submit and Approve with Test. Depending on your user rights, you might have to wait until someone else at your organization approves your changes.
7. A pop-up modal will appear titled 'Preview and start the test'.
8. Go through the 3 tabs at the top of the pop-up modal: Preview, Variables and Objective, and toggle settings as needed.
9. In the Preview tab, adjust all the settings in this window based on your hypothesis of the test.
10. Under Variables, choose your confidence level and the % of visitors you want exposed to the test.
11. In the Objective tab, select a goal that you would like to test this page to reach and then select how you would like the test to choose a winner.
12. Choose the minimum and maximum duration for the test and select the Start test button.

---

Page Tests

A page test compares one page's content against another.

1. In the Sitecore launchpad, select Experience Optimization.
2. Select the Create Page Test button.
3. Choose the original page that you want to test in the Sitecore content tree that has appeared.
4. Choose the language of the page that you want to test in the Language dropdown to the right. Both pages must be in the same language when running the test.
5. Click the select button.
6. Under ‘Pages to test against’ click the plus button.
7. Select the ‘Use existing page’ option.
8. Select the other page that you want to test against in the content tree that has appeared.
9. Click Select.
10. You can continue to add pages to test against each other if you would like, by selecting the plus button and repeating the steps above.
11. When you’ve finished adding the pages you’d like to test, click ‘Review and Start’ at the top of this window.
12. Adjust all the settings in this window based on your hypothesis of the test.
13. Select a goal that you would like to test this page to reach under the Test objective dropdown menu, and then select how you would like the test to choose a winner.
14. Choose the minimum and maximum duration for the test and select the Start test button.

---

Component Tests

A component test allows you to test different combinations of components.

1. In the Sitecore launchpad, select Content Editor.
2. Select the page that you want to run the component test on in the content tree.
3. Select Publish in the top navigation, and then Experience Editor.
4. Select the component that you would like to test on the page.
5. In the floating menu that has appeared, click the ‘test the component’ icon, that looks like two components stacked on top of each other.
6. Click the New Variation button in the top right.
7. Add a new name for the new component variation where Variation Name is written.
8. Select the 3 dots under Test Content Item, and select a rendering to test with.
9. Add any other variations you want to test against.
10. Press the OK button on the bottom right.
11. Save the page by clicking the Save icon.
12. Click the Sitecore launchpad button in the top left corner of the page.
13. Click on Content Editor and find the page that you just updated in the content tree.
14. In the top navigation select Review, then Submit and Approve with test. Depending on your user rights, you might have to wait until someone else at your organization approves your changes.

---

Viewing Your Tests

1. In the Sitecore launchpad, select Experience Optimization.
2. In the left navigation under Lists, select Active Tests.
3. Here you can view all tests that are currently active on your site.

---

Closing Thoughts

Multivariate and A/B tests are the first adaptive personalization forms that you'll want to master before you move onto the more complex stuff.

When testing don't forget:

• To only test one thing at a time.
• To always test every variant in the same language version.

I hope you found this blog useful and are able to get some interesting insights from your A/B and multivariate tests that allow you to convert more leads and reach your goals faster.

If you have any questions, you can post them in this Facebook group for Sitecore marketers and content authors, or reach out to me on Twitter @natmankowski.