How Multivariate Testing Works

Content variations appear to site visitors during an experiment. The page variation that a site visitor sees is randomly selected by the system. When a visitor navigates to a special page called the target, it is logged as a conversion. A conversion might be a donation, registration, sales lead, and so on. Conversions are tracked, and the variation that produces the most conversions is automatically promoted to the website.

Sometimes, this type of testing is called “A/B Testing”, but Multivariate Testing is far more than that. A/B testing provides only a few variations. With Multivariate Testing, the number of page combinations grows exponentially as the number of layout and content combinations increase.

IMPORTANT: The Multivariate Testing feature requires the use of PageBuilder templates. For more information, see Developing Wireframe Templates.

Also, the Multivariate testing feature is part of Ektron's Marketing Optimization Suite. For more information about the suite, including webinars, white papers, instant demonstrations, and so on, see Rapidly Test and Target to Optimize Results.

An Example of Multivariate Testing

In the example below, the marketing manager wants to know which of 3 ads produces the most visits to a registration page.

• The experiment page contains ads that link to a registration form.
• A target page contains the registration form.
• A conversion is a visits to the target page produced by each ad.

When the target number of conversions occurs, the variation achieving it automatically becomes the permanent page on the site. The optimum page runs on the website without further effort by the website Admin.

Using Widgets for Multivariate Testing

You use the following PageBuilder widgetWidgets are mini-applications that you place on a Web page using PageBuilder; a widget provides either specific functionality (calculators, search, social bars, etc.) or areas into which you can add content (content blocks, list summaries, collections, and so on).s for Multivariate Testing:

• Multivariate Experiment widget
• Multivariate Section widget
• Multivariate Target widget

The widgets look like this in the PageBuilder widget tray.

The following list describes each widget type.

Using PageBuilder Pages and Content

Before you create a Multivariate Experiment, you will need to have 2 Web pages: the Target page and the Experiment page.

Both pages are created with PageBuilder which lets you drag and drop widgets into your page. For more information about PageBuilder, see Creating Web Pages with PageBuilder. You may choose any type of widget for your pages.

Creating a Multivariate Experiment

Prerequisites

To run Multivariate Tests, you must be an Administrator or have the Multivariate Tester Role assigned. For more information see Defining Roles.

To create a Multivariate experiment:

1. Create a target page that contains the Target widget. See Also: Creating a Multivariate Target Page
2. Create an experiment page that contains the content variations with the Experiment and Section widgets. See Also: Creating a Multivariate Experiment Page
3. Run the experiment until it automatically ends or you stop it. See Also: Starting and Running an Experiment

Starting and Running an Experiment

When you create the Target Page and Experiment Page:

1. Select the PageBuilder page that contains the Multivariate Experiment widget.
2. Click File > Edit to see the Experiment widget.
3. On the Experiment widget, click the Start button.
   

   

4. After you click Start, you see the Stop and Show Report buttons.
   

   

When an experiment is running, visitors see a randomly-chosen page variation. The visitor continues to see this variation until the experiment is ended.

NOTE: During an experiment, testers may modify variations and content within the Multivariate Section widget.

Cookies in Multivariate Testing

When visitors view the experiment page, a cookie is set in the browser. This cookie indicates the specific variation seen by the visitor. From this point forward, if they next visit the target page, this cookie is read and the proper variation counter is increased.

A cookie persists forever or until the cookies are removed from the browser cache. The cookie gives a visitor a consistent page variation each time they revisit during the experiment.

Visitors that block cookies are not counted in conversions. Also, they see random page variations each time they visit the page.

Using a Target Page Outside of You Website

When your target page is outside of your website, you must create an intermediate page that contains the Target widget, and redirect the visitor to the outside address.

For example, to test visitors from your page to http://www.NFL.com:

1. Create a new PageBuilder page.
2. Add the target widget to it.
3. Add the code that will redirect the visitor from this page to www.NFL.com.
4. Create content with links to the target page you just created.

When the experiment is running, the experiment page links the visitor to this intermediate page and the conversion is counted. Then, the redirect takes the visitor to the external page.

Repeating an Experiment

To repeat an experiment after it has ended, you first must restore the history of this page. To restore history of a PageBuilder page, follow the steps in Restoring a Previous Version.

After you restore the previous version, you may edit the parameters of the experiment, modify the content, and run the experiment again.

Scheduling an Experiment

The PageBuilder page has the same properties as other content types, including the Schedule Property. To set the beginning and ending date of the page, edit the PageBuilder page and follow the procedure for setting content schedules as described in Scheduling Content.

Be sure to set the Schedule End Date to be long enough past the time you believe the experiment will automatically end.

Using Reports

Reports show how each variation has performed during the experiment. Click on the Show Report button in the Experiment widget to see the progress and results.

The bar graph shows the conversion percentage.

The numbers to the right of the bar graph show 2 values. The first value is the percent of conversions compared to the total number of visits to the experiment page. On variation #2 above, the “79%” means that 568 visitors to this variation have successfully reached the target page 451 times.

The numbers after the percentage show the actual conversations. The first number before the forward slash (/) indicates the total number of conversions during the experiment. The second number is the total number of times this variation was viewed by a visitor. Users that are logged in are counted as a page view to the experiment page but not counted on the target page as a conversion.

The following list describes how to use each button.

• Preview—Shows the page configurations for this specific variation. Notice that not only does the content in the Multivariate Section wizards change, but also the slider number changes too.
• Variation Enabled—When this appears, this variation is currently selected to run. Click this button to disable this variation.
• Variation Disabled—When this appears, this variation is currently not selected to run. Click this button to enable the variation.
• Promote—Click this button to promotes this specific variation to become the actual page seen by visitors. All other variations are disabled and the experiment is ended.

Using Multiple Section Widgets

You may create a PageBuilder page with more than one Section widget. This provides greater flexibility in helping to identify combinations of content and images that provide the best conversion results.

A page containing 2 Multivariate Section widgets is shown below.

The contents inside the Section widgets are part of the variations shown randomly during the experiment.

The Number of Variations Tested

The number of variations in a Multivariate Experiment grows exponentially as you add sections and variations.

For example, if you have 2 sections with 3 variations in each section, there are a total of 8 (23) pages to be tested. If you increase this to 4 variations each, there are 16 (24) pages.

Multivariate Target Widget Details

The target widget is placed on the page containing the content you wish people to find.

The target widget does not have any configurations and need only reside somewhere on the page. Use only one Target widget per PageBuilder page

Multivariate Experiment Widget Details

The Multivariate Experiment widget contains all of the information about the variations you use in an experiment. It also contains the Start/Stop button and the conversion limits. Use only one Experiment widget per PageBuilder page.

The image below is an example of the Multivariate Experiment widget before the experiment is started.

• Target—This is the content ID of the target page.
• Conversions—This is the number of conversions any variations must have before the experiment is stopped automatically.

Set the target page's ID number and the value of Conversions. When this value is reached, the experiment promotes the variation with the highest percent of success, and disables all other variations.

To start the experiment, click the Start button.

Below is an example of a Multivariate Experiment widget while an experiment is running. If desired, you can click the Stop button to stop the experiment.

The experiment widget also shows reports to indicate how each variation has performed during the experiment.

Click the Show Report button to see the conversion information during the experiment. For further information about reports, see Using Reports.

Multivariate Section Widget Details

The Multivariate Section widget contains the content variations for the experiment. By using a variety of content combinations, the experiment tests the best arrangement of content to accomplish the desired conversion goals. The content is presented randomly to the site visitor. You may use one or more Section widgets per PageBuilder page.

The following sample Web page contains the Multivariate Section widget.