Table of Contents
Beeswax Log Resources
Log Examples
Examples of various Beeswax log types can be found on the Beeswax GitHub repository.
Mapping Between ID and Inventory Source Names
The mapping between ID and Inventory Source names is available here.
Log Header Definitions
Detailed definitions of log headers, field lists, and data dictionaries can be accessed in the publicly available Beeswax Log File Header Definitions.
Log File Formats
Beeswax log files are provided in CSV format. For users who need JSON format, a CSV↔JSON converter can be used as Beeswax does not support JSON for batch logs.
Log Time Zone Information
All Beeswax logs include timestamps in both Eastern Time (ET) and Coordinated Universal Time (UTC). If the timestamp is not labeled with "_utc", it is in ET. Logs may be delayed slightly, and directories for hourly log dumps will reflect the UTC hour.
S3 Bucket Configuration for Logs
For Auction and Bid logs, a separate S3 bucket must be provided for each region (e.g., us-east-1, us-west-2, etc.). For Win and Conversion logs, a single S3 bucket can be used across regions.
Impression Notification Window
Beeswax waits 24 hours from the time of the auction to receive impression notifications. After this period, any impression notifications for that auction will be ignored. This typically affects less than 1% of total impressions.
Folder Structure for Log Files
Customizing the folder structure for different log types is possible. Discuss any folder restructuring with your Beeswax account manager for further details.
User ID Availability
In cases where the user has not been assigned a user ID or has not accepted third-party cookies, the user ID will be missing from the log entries.
Auction Logs
Contents of Auction Logs
Auction Logs contain all bid requests made to the Bidder instance, capturing key details of the auction process.
Uses of Auction Logs
Auction Logs are primarily used for analytics, helping users refine their bidding strategies. However, they cannot be used to create derivative segments using the user IDs included in the bid requests.
Segment ID Availability in Auction Logs
Auction Logs will only contain up to 600 characters for segment IDs, limited to the requests where a bid was returned. For a complete listing of all segment IDs for a given auction, users should reference Segment Logs and join them on the Auction ID.
Bid Logs
Delivery Window for Bid Logs
Bid Logs are delivered in near-real-time as small CSV files, typically containing a few thousand bids per file. These logs provide insight into bid performance.
Uses of Bid Logs
Bid Logs are helpful for calculating win rates across various dimensions, such as device type, geolocation, or creative. This data is valuable for optimizing bidding strategies, particularly for users who employ custom bidding agents.
Conversion Logs
Delivery Window for Conversion Logs
Like Bid Logs, Conversion Logs are delivered in near-real-time as CSV files, containing a few thousand records per file.
Attribution of Conversion Logs
Beeswax sends all conversions regardless of attribution status. Conversion Logs include both attributed and non-attributed conversions, providing a comprehensive view of campaign performance.
Uses of Conversion Logs
Conversion Logs enable users to analyze conversions at any time. Since these logs are unjoined, they need to be combined with Win Logs to gain a complete understanding of conversions and related campaign activity.
Batch Win Logs
Delivery Options for Batch Win Logs
Batch Win Logs are delivered either hourly or daily. These logs are joined, meaning conversion and click data are matched to impression data.
Delivery Time for Daily Batch Win Logs
Daily batch Win Logs are typically written to customers' S3 buckets between 6–7 AM ET (6 AM during daylight savings time, 7 AM otherwise).
Handling of Duplicate Auction IDs
Auction IDs may appear duplicated in Batch Win Logs when events are re-rendered from the browser or app. These events will only be de-duplicated at the batch level.
De-duplication of Impressions
Beeswax automatically de-duplicates impressions in Batch Win Logs using the auction_adgroup_id field.
Streaming Win Logs
Delivery Options for Streaming Win Logs
Streaming Win Logs are available in real-time via HTTP or Kinesis. These logs can be delivered in either JSON or Protocol Buffers (Protobuf) formats, with the option for joined or unjoined logs.
Difference Between Joined and Unjoined Streaming Win Logs
- Unjoined Logs: Each message contains a single event (Impression, Click, Activity, or Conversion).
- Joined Logs: Messages can contain multiple events, with auction, click, activity, and conversion data combined into a single message. As more information becomes available, new messages with updated data for the same auction are sent.
Region Requirements for Streaming Win Logs
- Kinesis: For unjoined logs, the Kinesis Stream must be located in the same region as the bidder instance. For joined logs, the Kinesis Stream must be in the US-EAST-1 region.
- HTTP: No region co-location requirements for HTTP endpoints.
Types of Events in Streaming Win Logs
Unjoined logs contain only one type of event per message (Impression, Click, Activity, or Conversion). In contrast, joined logs include a combination of events (e.g., RequestLogMessage + ImpressionLogMessage + ClickLogMessage).
Repetition of Auction Information in Joined Win Logs
As more data about a win event is received, Beeswax may send multiple messages with the same Auction ID, each containing updated information (e.g., adding click or activity data to the original impression data).
Time Threshold for Streaming Win Log Messages
Messages are posted to the stream indefinitely. However, if a message is received more than six hours after the auction, it will not include the RequestLogMessage information. This occurs in less than 1% of cases.
Segment Logs
Using Segment Logs for Audience Forecasting and Attribution
Segment Logs provide a complete list of matched first-party segments, which is essential for audience forecasting and attribution. These logs can be joined with Auction Logs on auction_id to extend segment visibility and improve targeting insights.
Data De-Duplication
Need for Data De-Duplication
Beeswax logs are designed for "at least once delivery," which can sometimes result in duplicate data entries. To ensure accuracy, logs should be de-duplicated using appropriate keys such as auction_adgroup_id (for bid logs) or conversion_id (for conversion logs).
De-duplication for Bid and Auction Logs
Because Beeswax's data infrastructure does not track historical records, duplication may occur. It is recommended to de-duplicate logs at the auction_adgroup_id level for bid logs and at the conversion_id level for conversion logs.
De-duplication in Bid Logs
Bid Logs should be de-duplicated on the auction_adgroup_id field, as multiple bids can be placed for the same auction by different line items.