Use * syntax for faster search
Page History
...
| Column | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||
| |||||||||||||||||
| Section | |||||||||||||||||
...
Overview
Customers can target The Experiments feature allows different line items to sets of be targeted to distinct, non-overlapping groups of users, allowing comparison of results to test performanceenabling performance comparisons between setups. Also known as “AA/B Testing” testing or “Randomized Control Studies”, the Experiments Randomized Control Trials (RCTs), this feature creates isolated user subsets of users, and ensures these users are only , ensuring each group is exposed to one Line Item setuponly one line item configuration at a time.
Use Cases
- Compare Bidding Strategies: You want to test two or more different Multiple bidding methods can be tested to see determine which gives yields the best resultresults. You can do this with This can be done by creating identical line items for targeting but , with audience splits to compare performance across different bidding strategies.
- Compare Multiple Creatives: You want to test two or more Different sets of creatives or landing pages can be tested to see identify which generates more conversions. You can do this This is achieved by creating one a separate line item per for each set of creatives and assigning a different distinct test group to each.
- Persistent Control Group: You want to set aside a A control group of creatives (e.g., PSAs) can be set aside to measure incremental lift. You can do this by This is done using the X Percent Hold-out Test Plans. Assign , where the PSA line item is assigned to the Control Group (a small portion of users) , and all other line items should be are assigned to the Test Group (large a larger portion of users).
...
How to use the Experiments feature:
Experiments can be enabled at the Campaign level, with Test Plans being assigned at Campaign-level and Test Groups at the Line Item level:
| Note |
|---|
Experiments and Ghost Bidding cannot be used in tandem on the same line items. |
...
| UI Steps | ||
|---|---|---|
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
|
...
|
...
|
...
| Note |
|---|
All campaigns currently running experiments have ID Type set to Standard. |
...
Reporting
Report Builder and Query Tool:
Conversion Report and Performance Report include the following fields:
- Test Group ID
- Test Group Name
- Test Plan ID
- Test Plan Name
When using this feature, our Beeswax reporting products will provide insights through the following:
Logs:
...
| Note |
|---|
Experiment ID type gets added to the Conversion and Performance reports in Report builder reporting. |
Report Builder and Query Tool:
Conversion Report and Performance Report include the following fields:
- Test Group ID
- Test Group Name
- Test Plan ID
- Test Plan Name
API
You can read more on our API support for Experiments feature here, and the new campaign_experiment_assignment service here.
FAQs
Q: How did you come up with the uniformly sized user groups?
A: Based on our research, these are the most prevalent use cases for running experiments.
Q: Why can't I apply tests to creatives instead of Line Items? It is a lot of extra work to create multiple Line Items.
A: When we were evaluating the different options, one important criterion is which implementation provides you with the most flexibility and allows the most use cases. Applying test groups on the line item level is the best option so far.
Q: What happens if the user does not have a cookie?
A: That user will not see anything within this campaign. The line items within the campaign have assigned test group, which makes the user part of a test group. If we encounter an unknown user, there will be no bidding on the request.
Q: What happens if the user deletes the cookies and it gets reset?
A: The user may get assigned to a different user group.
Q: Is IP, FreeWheel ID Network, and LiveRamp IDL available globally?
A: IP is available globally. However, FreeWheel ID Network and LiveRamp IDL for Experiments is only available to US users only.
Q: What should we use for Experiments using IP: IPv6 or IPv4? What happens if IP gets obfuscated?
A: IPv6 takes precedence when both types of IPs are present.
If bid request has obfuscated IPs, then the bid request is not eligible for line items using IP as an experiment type.
Q: What happens if a bid request does not contain the ID required for the experiment?
A: If an ID is not available on the bid request, the bid request is not eligible for campaign and line items running experiments.
Q: Can Experiments using LiveRamp IDL fallback like frequency cap?
A: No, experiment groups do not fallback.
Reporting fields relevant to Experiments feature includes:
| Field Name | Values | Log Files |
|---|---|---|
| experiment_user_index | Any value between 1 to 1000 randomly assigned to a user for the test group assignment |
|
| test_group_id | Test group assigned to the user who received the impression. | |
| test_plan_id | Test plan associated with the campaign. | |
| experiment_id_type | The ID type users selected on the campaign level to run the experiment
|
API
Read more on Beeswax API support for Experiments here.Q: Are Tapad IDs available for Experiments?
A: Not available currently.