Data Cardinality

The Watching That platform is architected as the pairing of live data apps connected to a continuous data pipeline.  

Instead of being a set of dashboards connected to a database/store that is updated periodically through batch movement of data sets (with some tooling off to the side), we have taken the design approach of building a data pipeline and connecting monitoring and optimisation apps to its head, allowing them to continuously access the freshest data possible (we call it First to Know).  

Because of this approach we can, and do, collect all available information about an ad/adbreak/viewing sessions and reconstitute for immediate use.  

This requires us to approach data storage in a different way to the traditional approaches.  We have two tiers of data streams based on their uses cases:

• Live Unbounded Data Feed- this is a short term data feed that is uncompressed and unbounded.  You use this to get precision results, access to individual sessions;
• Historical Reporting Data Feed - this is a long term data feed that is compressed by cardinality and bounded by dimensional intersections.  You use this to achieve long, detailed views of your data.

So whenever the Watching That team talk about rollups it pertains to the Historical data. To compress this data meaningfully we query it every 10 min, 1 hr, 1 day and 1 month using a predetermined dimensional filter (Insertion Order, Placement, Insertion Order x Placement, Insertion Order x Device Type etc) and we limit the result set to the top 225 sorted by the metric in question (impressions, ad request, errors etc).  

The Historical data feed is a record of your most active results determined by the dimensions that describe your data best.