---

Customers can target different line items to sets of non-overlapping users, allowing comparison of results to test performance. Also known as “A/B Testing” or “Randomized Control Studies”, the Experiments feature creates isolated subsets of users, and ensures these users are only exposed to one Line Item setup.

Use Cases

• Compare Bidding Strategies: You want to test two or more different bidding methods to see which gives the best result. You can do this with identical line items for targeting but with audience splits. 
• Compare Multiple Creatives: You want to test two or more sets of creatives or landing pages to see which generates more conversions. You can do this by creating one line item per set of creatives and assigning a different test group to each.
• Persistent Control Group: You want to set aside a control group of creatives (PSAs) to measure incremental lift. You can do this by using the X Percent Hold-out Test Plans. Assign the PSA line item to the Control Group (small portion of users), and all other line items should be assigned to the Test Group (large portion of users).

How to use the Experiments feature:

Experiments can be enabled at the Campaign level, with Test Plans being assigned at Campaign-level and Test Groups at the Line Item level:

1. Toggle on Campaign: Campaigns have the Experiment toggle.

• When the toggle is set to Yes, a new Experiment tab appears on the object that allows a Test Plan to be associated with the campaign. 
• When the toggle is set to No, the experiment tab does not appear. The toggle is set to "No" by default.
• If the toggle is set to Yes but no Test Plan is selected in the Experiment tab before closing the window, the toggle gets reset to No. It is recommended to set up the Experiment after you create both the Campaign and Line Item.

           2. Test Plans and Test Groups : Test Plans and Test Groups are pre-configured.

• There are 14 Test Plans available. Each test plan has a different split of users arranged into Test Groups, allowing users to pick the plan that fits their use case the best.
• Test Plans are associated at the campaign level and Test Groups are associated at the line item level.
• You can only assign test groups to line items whose campaigns have an assigned test plan.

3. Test Plan and Test Group Association:

a. From the Campaign object, you can associate Test Plans to a Campaign by selecting the Experiments Tab and selecting a Test Plan from the dropdown.

b. Once you select a Test Plan, select a Test Group for each of the line items within the campaign.

• All line items within the Campaign must have an assigned test group. If not, then you cannot save the associations.
• Multiple line items can be assigned to the same test group. 
• You do not need to select all test groups.

c. Once a Test Plan has been selected, all newly created line items show an Experiments section in their Overview page.

4. IPs, FreeWheel ID Network, and LiveRamp IDL: Users can run experiments on all devices using IPs, FreeWheel ID Network, LiveRamp IDL, especially for CTV/OTT devices. When running experiments, users can split test across all devices (for a person or entire household). 

In the Campaign > Experiment section, select an  ID Type from the dropdown to run your experiment. 

Experiment ID Type Options

Reporting

When using this feature, our reporting products will provide insights through the following:

Logs: 

Fields relevant to Experiments feature includes:

Report Builder and Query Tool: 

Conversion Report and Performance Report include the following fields:

• Test Group ID
• Test Group Name
• Test Plan ID
• Test Plan Name

API

You can read more on our API support for Experiments feature here, and the new campaign_experiment_assignment service here.

FAQs

Q: How did you come up with the uniformly sized user groups?
A: Based on our research, these are the most prevalent use cases for running experiments. 

Q: Why can't I apply tests to creatives instead of Line Items? It is a lot of extra work to create multiple Line Items.
A: When we were evaluating the different options, one important criterion is which implementation provides you with the most flexibility and allows the most use cases. Applying test groups on the line item level is the best option so far.

Q: What happens if the user does not have a cookie?
A: That user will not see anything within this campaign. The line items within the campaign have assigned test group, which makes the user part of a test group. If we encounter an unknown user, there will be no bidding on the request. 

Q: What happens if the user deletes the cookies and it gets reset?
A: The user may get assigned to a different user group. 

Q: Is IP, FreeWheel ID Network, and LiveRamp IDL available globally?
A: IP is available globally. However, FreeWheel ID Network and LiveRamp IDL for Experiments is only available to US users only.

Q: What should we use for Experiments using IP: IPv6 or IPv4? What happens if IP gets obfuscated?
A: IPv6 takes precedence when both types of IPs are present. 
If bid request has obfuscated IPs, then the bid request is not eligible for line items using IP as an experiment type.

Q: What happens if a bid request does not contain the ID required for the experiment?
A: If an ID is not available on the bid request, the bid request is not eligible for campaign and line items running experiments.

Q: Can Experiments using LiveRamp IDL fallback like frequency cap?
A: No, experiment groups do not fallback. 

Q: Are Tapad IDs available for Experiments?
A: Not available currently.