Table of Contents
FAQs
- Q: Where can I find examples of your Logs?
A: https://github.com/BeeswaxIO/beeswax-api/tree/master/beeswax/logs/batch - Q: Where can I find the mapping between ID and Inventory Source names?
A: This can be found here: https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax/openrtb/openrtb_common.proto#L828 - Q: Do you have definitions of the headers?
A: Yes. Column definitions, field lists and a data dictionary can be found in the is publicly-accessible directory on Github: Beeswax Log File Header Definitions - Q: Can we receive your batch logs in JSON?
A: Unfortunately not, we only support CSV. You can use a CSV↔JSON converter. - Q: What time zone are the logs in? What about for wins in US-West and EU-West?
A: All logs contain both ET and UTC timestamps. Timestamps in UTC will be appended with "_utc" at the end. If this is not appended, then the field is in ET. If you have set up logs to dump into a different directory for each hour of the day it should be noted that these hours are the UTC hour. Some logs are emitted on a delay, so these file paths may appear to be behind, but this is intentional.
- Q: Do i need to create a region specific S3 bucket for your batch logs?
A: For Auction and Bid logs you must give us an S3 bucket per region (e.g. us-east-1, us-west-2, ap-northeast-1 or eu-west-1). For Win and Conversion logs, you can give us a single S3 bucket across regions. For example,
Case 1: if the bidder instance is in US-WEST-2, then s3 bucket for Auction and Bid logs needs to be in US-WEST-2
Case 2: If bidder instance is set up in both US-WEST-2 and US-EAST-1 then, customer should provide S3 bucket in US-WEST-2 and a separate bucket in US-EAST-1
- Q: How long will Beeswax wait for an impression notification?
A: 24 hours from the time of the auction. After this time, Beeswax will ignore any impression notifications for a given auction (note, this should only affect <1% of total impressions). - Q: Are we able to change the folder structure of the different types of logs we receive?
A: Yes, please discuss this with your account manager. - Q: Why is the user ID missing on some of my log file entries?
A: The user did not have a user ID already assigned to them, and the user also did not accept third party cookies, so we were unable to issue them a user ID. - Q: Do you offer the logs in Parquet columnar format (instead of csv)?
A: Yes, see Parquet Support for more information.
Auction Logs
- Q: Do Auction Logs contain all bid requests into a Bidder instance or all requests into a Bidding Agent?
A: Auction Logs will contain all bid requests made to your Bidder instance. - Q: What can Auction Logs be used for?
A: Auction Logs can be used for analytics. You can use the data to inform future bidding strategies, but you cannot create derivative segments using the user IDs that come through on the bid requests. - Q: Will Auction Logs contain all segment IDs of my users by default?
A: No, by default Auction Logs will only contain segment IDs for requests where a bid was returned; furthermore, this listing will be limited to 600 characters. In order to see the full listing of relevant segments, you can use the Segment Logs and join the two on Auction ID.
Bid Logs
- Q: What's the delivery window for bid logs?
A: We deliver bid logs in near-real-time as small .csv files--usually, a few thousand bids per file. - Q: What are bid logs used for?
A: You can use bid logs to calculate win rates on any dimension that comes through on the bid & win logs, which you can then use to improve your bidding strategies. Often, clients that use a custom bidding agent will log the bids themselves.
Conversion Logs
- Q: What's the delivery window for conversion logs?
A: We deliver conversion logs in near-real-time as small .csv files--usually, a few thousand bids per file. - Q: Do you only send attributed conversions?
A: No, we send all conversions, regardless of whether they are attributed based on the impression & click windows. - Q: What are conversion logs used for?
A: Conversion logs allow clients to analyze any conversions that come in at any time. These logs are unjoined, and will need to be joined by the client with the win logs.
Batch Win Logs
- Q: What are the delivery options for batch win logs?
A: In batch, either hourly or daily. Batch files are always joined, which means conversion and click data is matched against impression data. - Q: Does viewability data from IAS and/or Moat always come through in win logs?
A: This can vary slightly dependent on when we receive the log files from the provider. typically IAS delivers data once per day, whenever they send their feed. Meanwhile, Moat delivers data to Beeswax hourly. - Q: What time does Beeswax write daily batch win logs to customer S3 buckets?
A: Between 6-7am ET (6am during daylight savings time, 7am during non-DST periods) - Q: Why are there duplicate auction_adgroup_id's?
A: These events are from the browser/app, so these may be re-rendered from the browser/app. Auction IDs will only be de-duped in batch logs. - Q: How do you deduplicate impressions?
A: Beeswax deduplicates all impression events using auction_adgroup_id in batch logs.
Streaming Win Logs
- Q: What are the delivery options for streaming win logs?
A: Win logs are available for streaming (real-time) via HTTP or Kinesis in either JSON or Protocol Buffer (Protobuf) formats. Streaming logs can be joined or unjoined. - Q: What are the differences between Real-time (Streaming) Joined and Unjoined Logs?
A: Unjoined logs will never contain information about the auction associated with the impression. Joined logs will match auction data, as well as click, activity and conversion data, back to the won Impression data in a single message.Unjoined logs will send a separate “unjoined” message for any Impression, Click, Activity or Conversion that occurs for a given auction_adgroup_id. In joined logs, as more information is received about a win event, a message will be sent with new information appended to the previously sent message for that Auction ID. This means new messages may repeat data from the same Auction ID that have already been logged. - Q: Are there AWS region requirements for Streaming Win Logs?
A: Kinesis: For unjoined logs, a Kinesis Stream must be co-located in the same region as the bidder instance. For joined logs, the Kinesis Stream must be co-located in US-EAST-1. HTTP: There are no requirements for colocation for an HTTP endpoint. - Q: What types of events will I receive in my Streaming Win Logs?
A: Joined and Unjoined Logs will receive slightly different sets of events with different behaviors:
| Unjoined Win Logs | Joined Win Logs |
Each message will contain only ONE of the following:
| Each message can contain one or many of the following Messages joined together:
|
- Q: Why do I receive the same information about an auction multiple times in Joined Win Logs?
A: As more information about a win event is received, Beeswax will re-send the previously received messages with additional information populated alongside. (e.g, Clicks, Activities) This can result in multiple messages with the same Auction ID.For example, given a single Auction ID you may receive a new POST request with each of the following contents as new messages come in:- RequestLogMessage + ImpressionLogMessage
- RequestLogMessage + ImpressionLogMessage + ClickLogMessage
- RequestLogMessage + ImpressionLogMessage + ClickLogMessage + ActivityLogMessage
- Q: What is the time threshold after a Bid Request where I will no longer receive messages about that auction?
A: Beeswax will continue to post messages to your stream about auctions indefinitely regardless of when they are received. However, messages received by Beeswax over 6 hours after the auction will not contain RequestLogMessage information joined. Because most messages are received within this time window, this should occur less than 1% of the time.
Segment Logs
- Q:How can I use Segment Logs?
A: Standard Auction logs will limit the segment_id field to 600 characters, however Segment Logs will give you a complete listing of matched 1st party segments. By joining Auction and Segment Logs on auction_adgroup_id (as the key), you will extend your ability to do Audience Forecasting and Attribution. Both these use-cases require being able to see all segments on a given auction.
Data De-Duplication
Beeswax Data Infrastructure uses an "at least once delivery" design pattern to ensure all events are eventually delivered to customers. In certain scenarios, this may mean that duplicative data is sent in logs to customers. We always recommend de-duplicating your log-level data on auction_adgroup_id in the case of bid logs or conversion_id in the case of conversion logs.
Q: Does de-duplication need to be done for bid and auction logs?
A: De-duplication will need to be done as they are emitted from a stateless service that does not store or track historical records that were passed through it, so it does not inherently de-dupe within itself.
Q: Why do bid logs need to be de-duped on auction_adgroup_id?
A: In the case of bid logs, we recommend de-duping by auction_adgroup_id, as multiple bids can be submitted for the same auction_adgroup_id by different line items.