How Multivariate Testing Works

Content variations appear to site visitors during an experiment. The page variation that a site visitor sees is randomly selected by the system. When a visitor navigates to a special page called the target, it is logged as a conversion. A conversion might be a donation, registration, sales lead, and so on. Conversions are tracked, and the variation that produces the most conversions is automatically promoted to the Web site.

Sometimes, this type of testing is called “A/B Testing”, but Multivariate Testing is far more than that. A/B testing provides only a few variations. With Multivariate Testing, the number of page combinations grows exponentially as the number of layout and content combinations increase.

IMPORTANT: The Multivariate Testing feature requires the use of PageBuilder templates. For more information, see Building Pages.

Also, the Multivariate testing feature is part of Ektron's Marketing Optimization Suite. For more information about the suite, including webinars, white papers, instant demonstrations, and so on, see http://ektron.com/marketingoptimization/.

An Example of Multivariate Testing

In the example below, the marketing manager wants to know which of 3 ads produces the most visits to a registration page.

• The experiment page contains ads that link to a registration form.
• A target page contains the registration form.
• A conversion is a visits to the target page produced by each ad.

When the target number of conversions occurs, the variation achieving it automatically becomes the permanent page on the site. The optimum page runs on the Web site without further effort by the Web site Admin.

Using Widgets for Multivariate Testing

You use the following PageBuilder widgets for Multivariate Testing:

• Multivariate Experiment Widget
• Multivariate Section Widget
• Multivariate Target Widget

The widgets look like this in the PageBuilder Widget tray.

The following table describes each widget type.

Widget	Page Used	Description
Multivariate Target Widget	Target Page	When a page view occurs on a page containing this widget, the conversion count is increased. See Multivariate Target Widget Details
Multivariate Experiment Widget	Experiment Page	This controls the experiment. Settings include the target content number, start/stop button and the Report hide/show button. This is only available when logged in and editing the PageBuilder experiment page. See Multivariate Experiment Widget Details
Multivariate Section Widget	Experiment Page	This widget lets you drag and drop various content widgets into it. These produce the variations used during the experiment. A slider control selects each specific content used in the variations. More than one Multivariate Section widget can be used on a page. See Multivariate Section Widget Details

Using PageBuilder Pages and Content

Before you create a Multivariate Experiment, you will need to have 2 Web pages: the Target page and the Experiment page.

Both pages are created with PageBuilder which lets you drag and drop widgets into your page. For more information about PageBuilder, see Creating Web Pages with PageBuilder. You may choose any type of widget for your pages.

Creating a Multivariate Experiment

NOTE: Prerequisites: To run Multivariate Tests, you must be an Administrator or have the Multivariate Tester Role assigned. For more information see Defining Roles.

To create a Multivariate experiment:

1. Create a target page that contains the Target Widget. See Also: Creating a Multivariate Target Page
2. Create an experiment page that contains the content variations with the Experiment and Section widgets. See Also: Creating a Multivariate Experiment Page
3. Run the experiment until it automatically ends or you stop it. See Also: Starting and Running an Experiment

Creating a multivariate target page

Creating a Multivariate Target Page

The target page is the Web page you wish your visitors to view. It contains the Multivariate Target Wizard. Visitors arrive here by clicking links on the experiment page.

To create the target page:

1. Create or edit the PageBuilder page with the target content.
2. Note and record the ID of the content number on this page. (For example: 948.) This ID number is used in Step 3 below Creating a Multivariate Experiment Page.
3. Add the Multivariate Target widget to any location on the page.
   

   

4. Publish this page

After creating the Target Page, create the Multivariate Experiment Page.

Creating a multivariate experiment page

Creating a Multivariate Experiment Page

The Experiment page contains:

• the Experiment Widget
• the Section Widget(s)
• the content variations

The following illustration shows the basic steps to create an experiment page with Multivariate Test widgets.

To create a multivariate experiment page:

1. Create or edit the PageBuilder page you wish to use during the experiment. See Also: Creating Web Pages with PageBuilder
2. Place the Multivariate Experiment widget in any location on your Web page. Do not worry about the appearance of this widget. Multivariate Test Widgets do not appear on a Web page when you are not logged in. Only the content shows. For more information, see Multivariate Experiment Widget Details
3. Enter the Target Page Content Id number. (For example, 948.) (See step 2 of Creating a Multivariate Target Page) To help find the content, browse the content folders by clicking the Ellipsis button.
   

   

4. Enter the Conversions before promote value. Setting this to 1000 tells the experiment to automatically end when one variation has reached 1000 conversions.
   

   

5. Place at least one Multivariate Section widget into the location where your content variations should appear. For more information, see Multivariate Section Widget Details
6. Drag any type of content widget into the Multivariate Section widget.
7. Configure the content widget with the appropriate settings. (You may use any widget inside the Multivariate Section Widget. The Content Widget was chosen for the purposes of this example.)
8. Add more variations by clicking the Add Variations button () and repeat steps 6 and 7. You now have variation 2 of 2.
   

   

9. If you wish to add another section and variations, go back to step 5 and continue. The image below shows a completed experiment page that has 2 Section Widgets.
   

   

10. Publish the PageBuilder page. Now you are ready to run the experiment.

Starting and Running an Experiment

When you create the Target Page and Experiment Page:

1. Select the PageBuilder page that contains the Multivariate Experiment Widget.
2. Click File > Edit to see the Experiment Widget.
3. On the Experiment Widget, click the Start button.
   

   

4. After you click Start, you see the Stop and Show Report buttons.
   

   

When an experiment is running, visitors see a randomly-chosen page variation. The visitor continues to see this variation until the experiment is ended.

NOTE: During an experiment, testers may modify variations and content within the Multivariate Section widget.

Displaying reports during an experiment

Displaying Reports During an Experiment

Click the Show Reports button to see the experiment’s progress. The reports shows:

• the percentage of conversions compared to visits
• the number of conversions
• the total number of visits to each variation

Click the Preview button () to see the page variation associated with the progress bar.

For more information on using and interpreting reports, see Using Reports.

Enabling or disabling a variation

Enabling or Disabling a Variation

At any time during the experiment, you may disable any variation. Do this by clicking the () button which currently shows the variation is enabled. The variation then becomes disabled and cannot be viewed by site visitors.

When a variation is disabled, visitors who have seen this page no longer see it but see another variation instead.

You may re-enable a variation by pressing the () button, which currently shows the variation as disabled. The variation then becomes enabled and visitors can view it.

Stopping the experiment

Stopping the Experiment

At any time during the experiment, click the Stop button on the Experiment Widget.

This suspends the experiment. Counters stop incrementing, even though visitors continue to see the page variations.

While the experiment is stopped, cookies are neither set nor read. For more information about cookies, see Cookies in Multivariate Testing. When you stop an experiment, the Multivariate Section Widget remains on the experiment page.

You may restart the experiment again by pressing the Start button. All counters are reset to 0 and restarted.

Automatically ending the experiment

Automatically Ending the Experiment

When one variation reaches the target number of conversions, it automatically becomes the page variation all visitors see. All other variations are disabled.

Manually ending the experiment

Manually Ending the Experiment

During the experiment, you may click on the Promote button () of the specific variation you wish to promote. This ends the experiment, and the selected variation becomes the page variation all visitors see. All other variations are disabled. When the experiment automatically or manually ends, the Multivariate Section Widget is removed from the experiment page.

Cookies in Multivariate Testing

When visitors view the experiment page, a cookie is set in the browser. This cookie indicates the specific variation seen by the visitor. From this point forward, if they next visit the target page, this cookie is read and the proper variation counter is increased.

A cookie persists forever or until the cookies are removed from the browser cache. The cookie gives a visitor a consistent page variation each time they revisit during the experiment.

Visitors that block cookies are not counted in conversions. Also, they see random page variations each time they visit the page.

Using a Target Page Outside of You Web Site

When your target page is outside of your Web Site, you must create an intermediate page that contains the Target Widget, and redirect the visitor to the outside address.

For example, to test visitors from your page to http://www.NFL.com:

1. Create a new PageBuilder page.
2. Add the target widget to it.
3. Add the code that will redirect the visitor from this page to www.NFL.com.
4. Create content with links to the target page you just created.

When the experiment is running, the experiment page links the visitor to this intermediate page and the conversion is counted. Then, the redirect takes the visitor to the external page.

Repeating an Experiment

To repeat an experiment after it has ended, you first must restore the history of this page. To restore history of a PageBuilder page, follow the steps in Restoring a Previous Version.

After you restore the previous version, you may edit the parameters of the experiment, modify the content, and run the experiment again.

Scheduling an Experiment

The PageBuilder page has the same properties as other content types, including the Schedule Property. To set the beginning and ending date of the page, edit the PageBuilder page and follow the procedure for setting content schedules as described in Scheduling Content.

Be sure to set the Schedule End Date to be long enough past the time you believe the experiment will automatically end.

Using Reports

Reports show how each variation has performed during the experiment. Click on the Show Report button in the Experiment Widget to see the progress and results.

The bar graph shows the conversion percentage.

The numbers to the right of the bar graph show 2 values. The first value is the percent of conversions compared to the total number of visits to the experiment page. On variation #2 above, the “79%” means that 568 visitors to this variation have successfully reached the target page 451 times.

The numbers after the percentage show the actual conversations. The first number before the forward slash (/) indicates the total number of conversions during the experiment. The second number is the total number of times this variation was viewed by a visitor. Users that are logged in are counted as a page view to the experiment page but not counted on the target page as a conversion.

The following list describes how to use each button.

• Preview—Shows the page configurations for this specific variation. Notice that not only does the content in the Multivariate Section wizards change, but also the slider number changes too.
• Variation Enabled—When this appears, this variation is currently selected to run. Click this button to disable this variation.
• Variation Disabled—When this appears, this variation is currently not selected to run. Click this button to enable the variation.
• Promote—Click this button to promotes this specific variation to become the actual page seen by visitors. All other variations are disabled and the experiment is ended.

Using Multiple Section Widgets

You may create a PageBuilder page with more than one Section Widget. This provides greater flexibility in helping to identify combinations of content and images that provide the best conversion results.

A page containing 2 Multivariate Section Widgets is shown below.

The contents inside the Section Widgets are part of the variations shown randomly during the experiment.

The Number of Variations Tested

The number of variations in a Multivariate Experiment grows exponentially as you add sections and variations.

For example, if you have 2 sections with 3 variations in each section, there are a total of 8 (23) pages to be tested. If you increase this to 4 variations each, there are 16 (24) pages.

Multivariate Target Widget Details

The target widget is placed on the page containing the content you wish people to find.

The target widget does not have any configurations and need only reside somewhere on the page. Use only one Target Widget per PageBuilder page

The target page

The Target Page

The target (or results) page contains the Multivariate Target widget.

Whether you arrive at the target page by browsing to the variations page first or by browsing to the target page directly, the views are counted with the following exceptions:

• if you are logged in
• if you have not first visited the experiment page to receive the cookie
• if you have cleared the cookies before visiting the target page

NOTE: Users that are logged in are counted as a page view of the experiment page but not counted on the target page as a conversion.

Multivariate Experiment Widget Details

The Multivariate Experiment Widget contains all of the information about the variations you use in an experiment. It also contains the Start/Stop button and the conversion limits. Use only one Experiment Widget per PageBuilder page.

The image below is an example of the Multivariate Experiment Widget before the experiment is started.

• Target—This is the content ID of the target page.
• Conversions—This is the number of conversions any variations must have before the experiment is stopped automatically.

Set the target page's ID number and the value of Conversions. When this value is reached, the experiment promotes the variation with the highest percent of success, and disables all other variations.

To start the experiment, click the Start button.

Below is an example of a Multivariate Experiment widget while an experiment is running. If desired, you can click the Stop button to stop the experiment.

The experiment widget also shows reports to indicate how each variation has performed during the experiment.

Click the Show Report button to see the conversion information during the experiment. For further information about reports, see Using Reports.

Automatic promotion

Automatic Promotion

Automatic promotion occurs when the target number of conversions occurs on one of the variations. For more information see Automatically Ending the Experiment.

Manual promotion

Manual Promotion

During an experiment, you may choose a specific variation to be the page that all visitors see and end the experiment. For more information see Manually Ending the Experiment.

Multivariate Section Widget Details

The Multivariate Section widget contains the content variations for the experiment. By using a variety of content combinations, the experiment tests the best arrangement of content to accomplish the desired conversion goals. The content is presented randomly to the site visitor. You may use one or more Section Widgets per PageBuilder page.

The following sample Web page contains the Multivariate Section widget.

Variation selection slider

Variation Selector Slider

The slider bar at the top of the Multivariate Section widget is used to select the variation.

The sample Web pages below demonstrate the slider control for variation 1/2 and 2/2.

Move the slider to see each variation.

Click Add Variation () to add another variation. Drag and drop the content widget into this section. After configuring it, this content widget is now one of the variations used in the experiment cycle.

Progress bar

Progress Bar

During the experiment, statistics are collected on:

• the number of page visits to the target page
• the number of page visits to a specific variation

The experiment progresses until it is either stopped or ended. For more information, see Using Reports.