we were able to construct a baseline that correlates better with reality and is less affected by the noise within the data.
The common way to measure the performance of TV spots, is to measure your website in the minutes following a commercial and comparing these results to the performance that we expected without a commercial. This process is called benchmarking. Your first step would be to measure the amount of sessions your website has after the commercial. The next step is to create a benchmark, also called a baseline to compare your results with.
The simplest method would be to measure the amount of sessions on your website before you aired the commercial and take that number as the baseline. While this solution may sound intuitive and adequate in theory, in reality it is just not accurate enough. The first problem is that your benchmark is static. You assume that it won’t change during the period after your commercial up to the point you stop measuring.
To fix this problem you can use two measure points instead of one. In addition to the measure point before airing the commercial you now also measure the point after the effect of the commercial has faded. Now you can simply draw a line from the first measure point to the second and use this line as your benchmark. The problem here is identifying when the effect of your commercial has faded. We will discuss this problem in a future blog.
Another problem is that this benchmark is based on a short snapshot of your website’s traffic. In such a short timeframe, sudden variations in your website visits can occur that will distort the baseline calculation. So the measured baseline may no longer be an accurate representation. This can result in significant higher or lower measured uplifts than what is really the case. So adding an additional measure point does not change anything. Now your baseline is simply based upon two potentially inaccurate points.
So why don’t we take the average traffic of each specific day and minute combination? Well that model is flawed because there is no such thing as an average Monday. Seasons, the weather, events, etc., can all heavily influence the baseline.
At Mediasynced, we have taken a more sophisticated approach in order to provide an accurate baseline. For each client, we build specific day models. These models are based on historical data and take several factors into consideration. They are then aligned with the actual data with the use of a y-intercept. Using this method we can construct a baseline that follows the natural curve we expect it to have, so based on the organic rhythm of the website throughout the day.
If we expect the traffic of your website to rise slightly from 16:10 to 16:20, then the constructed baseline for that time frame will also rise slightly. To increase the accuracy we have splitted our data up by device and repeated this process for each of those devices. Using these methods we were able to construct a baseline that correlates better with reality and is less affected by the noise within the data.
In the example below we see that the number of baseline sessions (real baseline) is oscillating around an average of 10 sessions per minute. During the commercial the baseline is starting a positive peak which the two simple baselines fail to capture. Our baseline on the other hand can correctly predict this pattern and recreates this peak as best as possible.
Hopefully, you’ve enjoyed reading our first blog post of this special series. Every month we will release a new article, so stay tuned!