Table of Contents



Overview

The Experiments feature allows different line items to be targeted to distinct, non-overlapping groups of users, enabling performance comparisons between setups. Also known as A/B testing or Randomized Control Trials (RCTs), this feature creates isolated user subsets, ensuring each group is exposed to only one line item configuration at a time.

Use Cases

How to use the Experiments feature:

Experiments can be enabled at the Campaign level, with Test Plans being assigned at Campaign-level and Test Groups at the Line Item level:

 Experiments and Ghost Bidding cannot be used in tandem on the same line items. 

Toggle on Campaign: When creating a campaign, toggle experiments on under the General tab.

Please note that deactivating a campaign with associated experiments will also remove the experiment(s) from the campaign.

  • When the toggle is set to Yes, a new experiment tab appears on the object that allows a Test Plan to be associated with the campaign. 

  • When the toggle is set to No, the experiment tab does not appear. The toggle is set to "No" by default.

  • If the toggle is set to Yes but no Test Plan is selected in the Experiment tab before closing the window, the toggle gets reset to No. It is recommended to set up the Experiment after creating both the campaign and line item.


Test Plans and Test Groups : Test Plans and Test Groups are pre-configured.

  • There are 15 Test Plans available. Each test plan has a different split of users arranged into Test Groups, allowing users to pick the plan that fits their use case the best.

  • Test Plans are associated at the campaign level and Test Groups are associated at the line item level.

  • Test groups can only be assigned to line items that have an assigned test plan in their campaigns.

Test PlanTest GroupsUser Index Range*
Five Percent Hold-out Plan (ID: 1)Control Group (5%) (ID: 1)
Test Group (95%) (ID: 2)		1-50, 51-1000
Ten Percent Hold-out Plan (ID: 2)Control Group (10%) (ID: 3)
Test Group (90%) (ID: 4)		1-100, 101-1000
Twenty Percent Hold-out Plan (ID: 3)Control Group (20%) (ID: 5)
Test Group (80%) (ID: 6)		1-200, 201-1000
Two-Way Split Plan (ID: 4)Group A (50%) (ID: 7)
Group B (50%) (ID: 8)		1-500, 501-1000
Three-Way Split Plan (ID: 5)Group A (33.3%) (ID: 9)
Group B (33.3%) (ID: 10)
Group C (33.4%) (ID: 11)		1-333, 334-666, 667-1000
Ten-Way Split Plan (ID: 6)Group A (10%) (ID: 12)
Group B (10%) (ID: 13)
Group C (10%) (ID: 14)
Group D (10%) (ID: 15)
Group E (10%) (ID: 16)
Group F (10%) (ID: 17)
Group G (10%) (ID: 18)
Group H (10%) (ID: 19)
Group I (10%) (ID: 20)
Group J (10%) (ID: 21)		1-100, 101-200, ..., 801-900, 901-1000
Three-Way Split with Fifty Percent
Hold-Out Plan (ID: 7)		Control Group (50%) (ID: 22)
Group A (25%) (ID: 23)
Group B (25%) (ID: 24)		1-500, 501-750, 751-1000
Twenty Five Percent Hold-out Plan (ID: 8)Control Group (25%) (ID: 25)
Test Group (75%) (ID: 26)		1-250, 251-1000
One Percent Hold-Out Plan (ID: 9)Control Group (1%) (ID: 27)
Test Group (99%) (ID: 28)		1-10, 11-1000
Three-Way Split with Ten Percent
Hold-Out Plan (ID: 10)		Control Group (10%) (ID: 29)
Group A (45%) (ID: 30)
Group B (45%) (ID: 31)		1-100, 101- 550, 551-1000
Three-Way Split with Eighty Percent
Hold-Out Plan (ID: 11)		Control Group (80%) (ID: 32)
Group A (10%) (ID: 33)
Group B (10%) (ID: 34)		1-800, 801-900, 901-1000
Three-Way Split with Ninety Percent
Hold-Out Plan (ID: 12)		Control Group (90%) (ID: 35)
Group A (5%) (ID: 36)
Group B (5%) (ID: 37)		1-900, 901-950, 951-1000
Four-Way Split Plan (ID: 13)Group A (25%) (ID: 38)
Group B (25%) (ID: 39)
Group C (25%) (ID: 40)
Group D (25%) (ID: 41)		1-250, 251-500, 501-750, 751-1000
Four-Way Split with Ten Percent
Hold-Out Plan (ID: 14)		Control Group (10%) (ID:42)
Group A (30%) (ID: 43)
Group B (30%) (ID: 44)
Group C (30%) (ID: 45)		1-100, 101-400, 401-700, 701-1000
Thirty Percent Hold-out Plan (ID: 15)Control Group (30%) (ID: 46)
Test Group (70%) (ID: 47)		1-300, 301-1000


User Index Range: Each Test Group is assigned a User Index Range, which indicates the size of a Test Group and represents how the overall Experiment audience has been split. The index value of a user displays which Test Group (or index range) they fall into, and therefore which messaging/strategies they have been exposed to. The User Index value is also made available in the win-logs and allows further analysis of test results.

Test Plan and Test Group Association:

a. From the Campaign object, associate Test Plans to a Campaign by selecting the Experiments Tab and selecting a Test Plan from the dropdown.

b. Once a test plan is chosen, select a Test Group for each of the line items within the campaign.

  • All line items within the Campaign must have an assigned test group. If not, then you cannot save the associations.
  • Multiple line items can be assigned to the same test group. 
  • You do not need to select all test groups.

c. Once a Test Plan has been selected, all newly created line items show an Experiments section in their Overview page.


IPs, FreeWheel ID Network, and LiveRamp IDL: Users can run experiments on all devices using IPs, FreeWheel ID Network, LiveRamp IDL, especially for CTV/OTT devices. When running experiments, users can split test across all devices (for a person or entire household). 

In the Campaign > Experiment section, select an  ID Type from the dropdown to run your experiment. 

All campaigns currently running experiments have ID Type set to Standard.

Experiment ID Type Options

Experiment ID Type

ID for assigning users to experiment groups

Standard (Default)


  • Bito for web 
  • MAID for app 
  • IP for CTV devices 


IP


  • IP 


FreeWheel ID Network Person
  • extends to all devices and IDs at the person/individual level.
FreeWheel ID Network Household
  • extends to all devices and IDs that belong to the people in a household.

LiveRamp Person


  • LiveRamp Person ID 


LiveRamp Household


  • LiveRamp Household ID




Reporting

Conversion Report and Performance Report include the following fields:

When using this feature, Beeswax reporting products will provide insights through the following:

Logs: 

Experiment ID type gets added to the Conversion and Performance reports in reporting.

Reporting fields relevant to Experiments feature includes:

Field NameValuesLog Files
experiment_user_indexAny value between 1 to 1000 randomly assigned to a user for the test group assignment
  • Bid logs
  • Win logs
  • Conversion logs (attributed)
  • Streaming logs
test_group_idTest group assigned to the user who received the impression.
test_plan_idTest plan associated with the campaign.
experiment_id_type

The ID type users selected on the campaign level to run the experiment

  • Standard
  • IP
  • LIVERAMP_PERSON
  • LIVERAMP_HOUSEHOLD

API

Read more on Beeswax API support for Experiments here.