--- title: "Experiments" canonical: "https://hub.freewheel.tv/space/BC/115507257/Experiments" format: markdown --- | | | --- | | #### Table of Contents
> Macro (toc) | --- # Overview Experiments allow different line items to be targeted to distinct, non-overlapping groups of users, enabling performance comparisons between setups. Also known as A/B testing or Randomized Control Trials (RCTs), this feature creates isolated user subsets, ensuring each group is exposed to only one line item configuration at a time. Experiments are enabled on campaigns. Test plans are assigned at the campaign level, and test groups are enabled at the line item level. You can use experiments to do the following: - Compare bidding strategies. - Multiple bidding methods can be tested to determine which yields the best results. This can be done by creating identical line items for targeting, with audience splits to compare performance across different bidding strategies. - Compare effectiveness of multiple creatives. - Different sets of creatives or landing pages can be tested to identify which generates more conversions. This is achieved by creating a separate line item for each set of creatives and assigning a distinct test group to each. - Set a persistent control group. - A control group of creatives (e.g., PSAs) can be set aside to measure incremental lift. This is done using the X Percent Hold-out test plans: A control line item is assigned to the control group (a small portion of users) and all other line items are assigned to the test group (a larger portion of users). For information on API support for experiments, see the API documentation [here](https://docs.beeswax.com/v0.5/docs/experiments). --- # Create an Experiment If you have not done so already, create the campaign and any associated line items to run the experiment with. For a full walkthrough of campaign or line item creation, see [Create a Campaign](https://freewheel-knowledge-hub.atlassian.net/wiki/spaces/BC/pages/113836735) or [Create a Line Item](https://freewheel-knowledge-hub.atlassian.net/wiki/spaces/BC/pages/115507203). After your campaign and line items are created, do the following: 1. Go to **Trafficking **> **Campaigns** > select a campaign's **Name & ID** to open the **Edit Campaign** page. 2. Navigate to the **General **tab and scroll down to the **Experiments **section. > ℹ️ Deactivating a campaign with associated experiments will also remove the experiments from the campaign. 3. Select the **Test Plan** to use. The test plan is associated at the campaign level. It determines the number of test groups and how users are divided into those groups. For more information on each test plan, see the** ****[Experiment Test Plan and Test Group Details](#Experiment-Test-Plan-and-Test-Group-Details)** section below. 4. Select the **ID Type **for your experiment. This determines how users are identified for test groups. The following table details how the ID type options work: | **Experiment ID Type** | **ID used for assigning users to experiment groups** | | --- | --- | | Standard (Default) | - Bito for web - MAID for app - IP for CTV devices | | IP Address | Device IP Address | | FreeWheel ID Network Person | Extends to all devices and IDs at the person/individual level. | | FreeWheel ID Network Household | Extends to all devices and IDs that belong to the people in a household. | | LiveRamp Person | LiveRamp Person ID | | LiveRamp Household | LiveRamp Household ID | 5. Assign a **Test Group **for each line item in the campaign. - All line items within the Campaign must have an assigned test group. - Multiple line items can be assigned to the same test group. - You do not need to select all test groups. > ℹ️ Once a Test Plan has been selected, all newly created line items show an Experiments section on their Overview page. 6. Save the experiment by selecting the **Save & Continue** button at the bottom right of the screen. --- # Experiment Test Plan and Test Group Details Fifteen different test plans are available. Each test plan has a different split of users arranged into test groups, allowing you to pick the plan that best fits your use case. Test plans are associated at the campaign level, and test groups are associated at the line item level. All available test plans and their test group splits are described in the following table: > ℹ️ Each test group is assigned a **User Index Range**, which indicates the size of a test group and represents how the overall experiment audience has been split. The index value of a user displays which test group (or index range) they fall into, and therefore which messaging/strategies they have been exposed to. The User Index value is also made available in the win-logs and allows further analysis of test results. | **Test Plan** | **Test Groups** | **User Index Range** | | --- | --- | --- | | Five Percent Hold-out Plan (ID: 1) | Control Group (5%) (ID: 1) Test Group (95%) (ID: 2) | 1-50, 51-1000 | | Ten Percent Hold-out Plan (ID: 2) | Control Group (10%) (ID: 3) Test Group (90%) (ID: 4) | 1-100, 101-1000 | | Twenty Percent Hold-out Plan (ID: 3) | Control Group (20%) (ID: 5) Test Group (80%) (ID: 6) | 1-200, 201-1000 | | Two-Way Split Plan (ID: 4) | Group A (50%) (ID: 7) Group B (50%) (ID: 8) | 1-500, 501-1000 | | Three-Way Split Plan (ID: 5) | Group A (33.3%) (ID: 9) Group B (33.3%) (ID: 10) Group C (33.4%) (ID: 11) | 1-333, 334-666, 667-1000 | | Ten-Way Split Plan (ID: 6) | Group A (10%) (ID: 12) Group B (10%) (ID: 13) Group C (10%) (ID: 14) Group D (10%) (ID: 15) Group E (10%) (ID: 16) Group F (10%) (ID: 17) Group G (10%) (ID: 18) Group H (10%) (ID: 19) Group I (10%) (ID: 20) Group J (10%) (ID: 21) | 1-100, 101-200, ..., 801-900, 901-1000 | | Three-Way Split with Fifty Percent Hold-Out Plan (ID: 7) | Control Group (50%) (ID: 22) Group A (25%) (ID: 23) Group B (25%) (ID: 24) | 1-500, 501-750, 751-1000 | | Twenty Five Percent Hold-out Plan (ID: 8) | Control Group (25%) (ID: 25) Test Group (75%) (ID: 26) | 1-250, 251-1000 | | One Percent Hold-Out Plan (ID: 9) | Control Group (1%) (ID: 27) Test Group (99%) (ID: 28) | 1-10, 11-1000 | | Three-Way Split with Ten Percent Hold-Out Plan (ID: 10) | Control Group (10%) (ID: 29) Group A (45%) (ID: 30) Group B (45%) (ID: 31) | 1-100, 101- 550, 551-1000 | | Three-Way Split with Eighty Percent Hold-Out Plan (ID: 11) | Control Group (80%) (ID: 32) Group A (10%) (ID: 33) Group B (10%) (ID: 34) | 1-800, 801-900, 901-1000 | | Three-Way Split with Ninety Percent Hold-Out Plan (ID: 12) | Control Group (90%) (ID: 35) Group A (5%) (ID: 36) Group B (5%) (ID: 37) | 1-900, 901-950, 951-1000 | | Four-Way Split Plan (ID: 13) | Group A (25%) (ID: 38) Group B (25%) (ID: 39) Group C (25%) (ID: 40) Group D (25%) (ID: 41) | 1-250, 251-500, 501-750, 751-1000 | | Four-Way Split with Ten Percent Hold-Out Plan (ID: 14) | Control Group (10%) (ID:42) Group A (30%) (ID: 43) Group B (30%) (ID: 44) Group C (30%) (ID: 45) | 1-100, 101-400, 401-700, 701-1000 | | Thirty Percent Hold-out Plan (ID: 15) | Control Group (30%) (ID: 46) Test Group (70%) (ID: 47) | 1-300, 301-1000 | --- # Reporting with Experiments The Conversion Report and Performance Report include the following fields related to experiments: - Test Group ID - Test Group Name - Test Plan ID - Test Plan Name ## **Experiments in Logs** > ⚠️ The Experiment ID type gets added to the Conversion and Performance reports. For more information on reports, see [Reporting](https://freewheel-knowledge-hub.atlassian.net/wiki/spaces/BC/pages/115343463). Reporting fields relevant to experiments include: | **Field Name** | **Description** | **Log Availability** | | --- | --- | --- | | experiment_user_index | A value between 1 to 1000 randomly assigned to a user for the test group assignment | - Bid logs - Win logs - Conversion logs (attributed) - Streaming logs | | test_group_id | The test group assigned to the user who received the impression. | | test_plan_id | The test plan associated with the campaign. | | experiment_id_type | The ID type users selected on the campaign level to run the experiment. Possible values include:
- Standard - IP - LIVERAMP_PERSON - LIVERAMP_HOUSEHOLD | --- # Training Resources > Macro (ui-button) > Macro (ui-button)