A/B test

With an A/B test, you can test a mailing in different variants on a small portion of the recipient list before you start the actual dispatch. The variant that performs best in the criteria you defined is then sent to the remaining recipients.

The variants (split mailings) usually differ only in few parameters from another. The most common variations include:

A/B test in Smart Campaigns

To perform a A/B test in Smart Campaigns, follow these steps:

  1. Drag an A/B test node from the left action area and drop it onto the desktop.

    Image: Draging an A/B test node onto the desktop

  2. Add a Recipients node and select one or more recipient lists.

    Image: Adding a Recipients node

  3. Connect the Recipients node to the A/B test node.

    Image: Connecting the Recipients node to the A/B test node

  4. Drag a message node (Email or Mobile Push) onto the desktop. Create a new message or edit an existing message.

    Image: Adding a message node

  5. Drag another message node of the same type onto the desktop. Select the message that you used for the first message node and change the content according to the criteria that you want to test.

    Image: Adding and editing another message node

  6. Add further message nodes if necessary and follow the description in point 5.
  7. Connect each message node to the A/B test node.

    Image: Connecting the message nodes to the A/B test node

  8. In the A/B Test node, optionally activate Activate Automatic Optimization and/or Send time optimization. For example, you can configure that the mailing with the best unique opening rate is automatically sent to the rest of the recipient list two days after the dispatch of the split mailings. For more information, see A/B test node and Configuring send time optimization in the A/B test node.

    Image: Configuring the A/B test node

  9. In the Edit menu  of the respective connection, you can set, among other things, how many recipients should receive the split mailing. For example, if you select 10% for two split mailings, the mailing will be sent to the remaining 80% of the recipient list after the A/B test has been completed. For more information, see A/B test node.

    Image: Configuring the connections to the message nodes

  10. To complete the editing, click Save and Close. After the successful validation, your A/B test campaign is ready to be activated.

A/B test in Mailings Classic

A split mailingVariation of a regular mailing where a subset of recipients is "split off". Also called split-run testing, A/B testing or bucket testing. Its purpose is to test various emails against each other to determine which version is best to use as the master mailing. in Mailing Classics is a variation of a regular mailing where a subset of recipients is "split off". You can edit every split mailing like a regular mailing and can also have a completely different look to the master mailing.

A split mailing is associated with a master mailing, even if the master is moved to another folder.

To create split mailings in Mailings-Classic, do the following:

  1. In the Mailings window, select the desired mailing – which is called the master mailing – and click Split. The Split dispatch window opens.
  2. You have two options.

    In the Splits area, you can see the new split and existing splits of the selected master mailing. You can edit some parameters in the Split section of the Split dispatch window, including:

    • Name. This name is used internally. By default, the split receives the same name as the master mailing.
    • Subject. Enter your desired variation to test different subject lines against one another.
    • Maximum number of recipients. Specify the number of recipients whom the split should be sent (relative to master mailing recipients, in absolute numbers or a percentage).
    • Target groups. Select up to three target groups for the split. See Target groups.
    • Sending date. Set the dispatch date for each split. See Dispatch order.
  3. Click Save settings. The split is displayed in the mailing list.

    In the list, splits appear indented below the master mailing they belong to. In the Type column, they are designated as Split.

Recipients

To perform an A/B testWith an A/B test, you can test a mailing in different variants on a small portion of the recipient list before you start the actual dispatch. The variant that performs best in the criteria you defined is then sent to the remaining recipients., during which the best option between split A and B is selected and sent out as master mailing, splits are typically sent to a subset of total recipients. Each split can be sent to an absolute number of recipients, or a percentage of total recipients. If percentage is selected, it is based on the number of master mailing recipients. This principle also applies if a split uses a recipient list other than the master mailing.

For selecting a subset of recipients from a recipient list:

  • Percentage. Enter a percentage in the Maximum recipients field. The number of recipients is based on the total number of master mailing recipients.

    Or, enter an absolute value. If the split contains the corresponding number of recipients, it is sent to exactly that number.

  • Random. Recipients are randomly selected from the list. If you do not use this option, emails are sent to split recipients according to their order on the list.

In both instances, recipients who receive one split email will not be sent another. Similarly, recipients who are on several lists only receive one email. Episerver Campaign monitors email addresses during dispatch to ensure that each recipient only receives one email (split or master mailing).

Recipient lists

In most cases, the same recipient lists are used for the master mailing and splits. However, for a split, you can use a recipient list other than master mailing.

The master mailing recipient list is the basis for calculating the number of split recipients.

To use different recipient lists for the master and split mailing, in the Split dispatch window, select off from the Automatically use the best split for the master mailing drop-down list.

If you are using different recipient lists in combination with recipient data-based target groups, be aware of potential conflicts between the different recipient lists. See Target groups.

Calculating recipients

The total number of recipients (master mailing and splits) is calculated every time before a mailing – including individual splits – is sent. The following illustration shows how subsets of recipients and the total number of recipients are calculated during dispatch.

Image: Calculating recipients

Recipient calculation

  • The Episerver image symbol signifies that a subset of the total is being created.
  • If a percentage is given for the number of split recipients, discrepancies between individual splits are possible, as cancellations and bounce overflows are taken into account for each case. This reduces the number of possible total recipients and the correspondingly calculated percentages.
  • The number of possible recipients also considers the master mailing's target groups. If a target group with the recipient list of the master mailing contains fewer addresses than the same target group with the recipient list of the split, the number of recipients to be sent emails is the number calculated for the target group and recipient list of the master mailing.
  • If the total recipient number is reached during the dispatch of a split, the subsequent splits and the master mailing are not be sent (see example 1 and example 2).

Target groups

Target groups used in the master mailing are inherited by the split mailings. You can add target groups to split mailings by linking with the target groups of the master mailing using AND logic.

The number of matches of a target group in the recipient list of the master mailing furthermore determines the maximum possible number of recipients for this target group. If the recipient list of a split contains more matching recipients for this target group, the total number of recipients sent an email will not exceed the number previously calculated on the basis of the target group and recipient list of the master mailing.

Master mailing vs. split target groups

Avoid target groups in the main and split mailings that exclude each other (for example, "Male" and "Female"), because no recipients will be selected for the split.

Note that the creation of target groups using recipient-based criteria can only apply when the corresponding criteria is also defined as a list box. Furthermore, if using a different recipient list for the split than for the master mailing, ensure that it incorporates the same data structure as the master mailing list – or at least that the list boxes for the master mailing target group can also be found in the split. The description of the corresponding list boxes must be an exact match, including capitalization.

Example

If a target group is used in the master mailing with the criteria Recipients with interest "Football", the recipient lists of both the master mailing and the split must contain a common corresponding field, such as Interests.

Dispatch

To send a split mailing in the Split window, click Start split or select the check box below Mailing schedule and specify a dispatch date using the calendar.

Dispatch order

Set the order of splits in the lower section of the Split mailing window via arrow keys to the right of each split. Splits associated with a master mailing are part of a hierarchy, which is determined by the order of the splits. This split hierarchy is referenced when calculating recipients of splits and the master mailing.

Split hierarchy vs. sending order

When individual splits are manually started, the split hierarchy applies, without regard for dispatch order. For example, if the last split in a list is sent first, the recipients for the split and master mailings are calculated on the basis of the overall split hierarchy.

Under certain circumstances, this may lead to a split being sent to fewer or even no recipients despite having started first. This is because the splits higher up on the list are allocated the complete number of recipients, and no recipients remain for the subordinate split. For more information, see Split mailing examples.

If the split dispatch is started automatically, the split mailings are sent in the order of the split hierarchy.

Optimizing and sending the master mailing

An A/B test lets you analyze each split's performance and send the best performing variation as a master mailing.

Episerver Campaign also offers an automatic optimization. To activate it, select one of the following criteria from the Automatically use the best split for the master mailing drop-down list. This criterion determines which split is chosen for the master mailing.

Which criterion you use depends on mailing type, the quality of recipient data, and your marketing goals.

There are three options to start the master mailing.

  • Save mailing and start it manually later. Start the master mailing manually by clearing the check boxes and selecting the default option when editing the mailing in the Options tab.
  • Sending date for the master mailing. Set a sending date for the master mailing and select the check box to activate this option. Use this option if you usually send your newsletter at a certain day and time.
  • Sending delay for the master mailing after the last split mailing has been sent. Set a delay between the sending of the last split mailing and the master mailing. Select the check box to activate this option.

    Use this option to get reliable opens and click rates based on the split mailings, because not all recipients open and click the mailing immediately upon receiving it.

Split mailing examples

The following examples to help you understand the split module by showing recipient calculations and their results. They illustrate the hierarchies and dependencies of the master mailing and splits.

In all examples, it is assumed that recipients are valid (that is, no one is blocked, has canceled, or exceeded the bounce limit).

Example 1

The master mailing and splits are sent to the same recipient list. A limit on the recipient number is set in the master mailing. The splits comprise A/B testing (split A against split B) with different target groups. The recipient list contains 100 recipients (40 male, 40 female, 20 unspecified).

Mailing Recipient list Max. number of recipients Target group Sent emails
Master mailing A 28% (= 28 recipients) 0
Split 1 A male 28
Split 2 A female 0

Result: The total number of recipients, as defined in the master mailing, was reached during the dispatch of split 1. No emails are sent to Split 2 and the master mailing.

Example 2

Master mailing and splits are sent to different recipient lists. A limit for the total number of recipients is defined in the master mailing. Recipient list A (master mailing) contains 200 recipients (60 male, 60 female, 80 unspecified). Recipient list B (splits) contains 100 recipients (40 male, 40 female, 20 unspecified).

Mailing Recipient list Max. number of recipients Target group Sent emails
Master mailing A 28% (= 56 recipients) - 0
Split 1 B male 40
Split 2 B female 16

Result: The total number of recipients is determined by recipient list A and the limit of recipients defined in the master mailing. Split 1, only sent to male recipients with no sending limit, is sent to male recipients in list A. Split 2 is subsequently sent to 16 female recipients from the same list. At this point, the total number of recipients limit is reached.

Example 3

The master mailing and splits are sent to the same recipient list. Recipient list A contains 100 recipients (20 male, 20 female, 60 unspecified). Three splits are created.

  • Split 1 has the target group "male"
  • Split 2 "female"
  • Split 3 has the target group "male OR female" (so gender must be specified).

The master mailing does not define a target group. Split 3 is to be sent first.

Mailing Recipient list Max. number of recipients Target group Sent emails
Master mailing A 60
Split 1 A 10% female 10
Split 2 A 10% male 10
Split 3 A 10% unspecified 0

Result: While the first two target groups are subsets of the master mailing target group, the target group for split 3 and the master mailing group exclude one another. To select recipients who did not specify gender, the target group of the master mailing should be "males OR females OR blank field".