Data Cardinality

How Watching That Uses Data

What are 'rollups'? How does Watching That ingest, compress, and store data? This article explains exactly how Watching That's approach to data differs from what you might be used to - and how that affects the data you see in the platform.

The Watching That platform is built as the pairing of live data apps connected to a continuous data pipeline.  

Most Business Intelligence software consists of a set of dashboards connected to a database/store, that is updated periodically through batch movement of data sets (with some tooling off to the side). But we have taken a different approach, and built a data pipeline and connected monitoring and optimisation apps to its head. This allowins the platform to continuously access the freshest data possible. 

We like to call this 'First to Know'.  

Because of this approach we collect all available information about an ad, adbreak or viewing session - and reconstitute it for immediate use.  

This requires us to approach data storage in a different way to the traditional approaches. We have two tiers of data streams based on their uses cases:

1. Live Unbounded Data Feed - this is a short term data feed that is uncompressed and unbounded. You use this to get precision results, and access to individual sessions.

2. Historical Reporting Data Feed - this is a long term data feed that is compressed and bounded by dimensional intersections. You use this to achieve long, detailed views of your data.

So whenever the Watching That team talk about 'rollups', it pertains to the Historical data [2]. 

To compress this data meaningfully we query it every 10 min, 1 hr, 1 day and 1 month using a predetermined dimensional filter (Insertion Order, Placement, Insertion Order x Placement, Insertion Order x Device Type etc). We limit the result set to the top 225 sorted by the metric in question (impressions, ad request, errors etc).  

The Historical data feed is a record of your most active results, determined by the dimensions that describe your data best.