Skip to main content

A/B testing

Uniform Optimize provides A/B testing capabilities that enable you to measure the effectiveness of personalization by exposing some visitors to personalization and other visitors to default content.

In Uniform, if an A/B test is active, each visitor is either:

  • Included in the test - Any component that has personalization rules assigned will display its default content to the visitor.
  • Excluded from the test - Personalization rules are executed as normal.

In both cases, the Uniform tracker captures personalization activity details. This means that even though a visitor who is included in a test does not see any personalized content, details about what the visitor would have seen is captured.

How it works#

A/B testing is fully integrated into Uniform's tracking and personalization processes:

  • Tests are configured in Sitecore.
  • The test configuration is exposed to the Uniform tracker along with the other tracker settings that are configured in Sitecore.

Uniform tracker is responsible for determining whether the visitor is included in a test. The tracker sets this value in the "test cookie" so the personalization engine can determine whether to show personalized content to the visitor.

The following diagram illustrates the logic the tracker uses to determine whether the test cookie is set:

Uniform personalization uses the test cookie to determine whether the content displayed to the visitor is personalized or is the default content.

The following diagram illustrates the logic the personalization engine uses to determine whether to run personalizations rules:

FAQ#

Can test settings be changed after a test has started?#

Yes. However, data already collected will not be changed. For example:

  • You have a test set to run during all of 2021. You change the test to run only during June 2021. Data collected from January through May 2021 will remain.
  • You have a test configured with a sample size of 15%. You change the sample size to 5%. Visitors who are already included in (or excluded from) the test will remain included (or excluded).

What is the purpose of "sample size" for a test?#

Each Uniform test has a sample size associated with it. The sample size determines how likely a visitor is to be included in a test. For example, if the sample size is set to 10, that any visitor has a 10% chance of being included in the test.

Can test sample sizes be changed?#

Yes. Uniform Optimize comes with several sample sizes predefined. Additional sample size options can be enabled by adding new items to the following folder:

sitecore > System > Uniform > Optimize > Testing > Sample Sizes

Is Uniform testing compatible with Sitecore Testing?#

Sitecore's testing functionality requires the Sitecore CD instance handle the visitor request. Since Uniform decouples page delivery from the Sitecore CD instance, Sitecore testing cannot be used with Uniform.

What if the Sitecore Xdb dispatcher is used?#

Still no. Sitecore testing requires that Sitecore handle the execution of tracking and personalization. Dispatching visitor activity to Sitecore Xdb means that the Sitecore CD instance is able to handle tracking, but personalization remains decoupled from the Sitecore CD instance.

Configuration#

A/B tests for Uniform Optimize are configured using Sitecore Content Editor.

  1. In Sitecore Content Editor, navigate to

    sitecore > System > Uniform > Optimize > Testing > Tests

  2. Create a new item using the template Uniform Test.

  3. Enter values for the following fields:

    • Enabled - A test must be enabled in order for it to be active.
    • Sample size - Determines how likely a visitor is to be included in a test. This value is a percentage.
    • Start date - An enabled test may be considered active if its start date is in the past. If the start date is not specified, start date is not considered as a factor when determining if a test is active.
    • End date - An enabled test may be considered active if its end date is in the future. If the end date is not specified, end date is not considered as a factor when determining if a test is active.
    • Sites - Select the Sitecore sites that the test will apply to.

  4. Save and publish the item.

Reporting#

The Uniform tracker captures details for every component that is personalized. One of the details captured indicates whether the visitor was included in a test or not.

The property isIncludedInTest is included in each personalization activity captured by the tracker. The following is an example of a personalization activity captured by the tracker when the visitor was included in a test:

{    "changes": {        "component": null,        "data": {            "before": "d804f0b8-1875-448c-a87f-5b1c8ed3aacb",            "after": "83ea06bd-5412-461b-b265-19eec5df1bff"        },        "hide": false    },    "component": {        "id": "66b1ef77-4402-41dc-ac9d-b84de6f6d7a1",        "name": "Section Teaser"    },    "page": {        "id": "472b46f8-59fd-4ade-9d5b-0c1d8f4654d9",        "name": "New Releases"    },    "rule": {        "id": "0ede8e8c-e9dc-44c8-9327-d857504e38a1",        "name": "Trusted advisor rule"    },    "isIncludedInTest": true}

Since Uniform Optimize does not have a centralized database that captures visitor behavior (such as personalization activity), reporting requires that the dispatch functionality be used. Dispatch is the process by which the Uniform tracker transmits tracking data to another system (called the "destination").

Dispatch makes it possible for tracking data from all visitors to be sent to a single destination. The destination's own reporting features can then be used to measure personalization performance. For example, if Google Analytics is the destination, Google Analytics reporting tools can be be used.