As the mobile industry moves to more granular ad targeting, the amount of data required to analyze and log has increased significantly. Advertisers now use numerous data points in their efforts to retarget and determine if they want to display an ad to a particular user. This type of targeting has brought several challenges, not the least of which is the amount of data being stored and processed.
Besides the additional time required to process and match up user data with their previously logged profiles, the total data points now being logged has expanded greatly. Now user geographic, demographic, device and performance data is being logged. All of this data creates challenges for how to store, aggregate, protect and search it quickly.
A single visit by a user could easily have 20 separate events logged and each logged event could have upwards of 30 to 50 different data points stored. Any significant traffic volume can easily move the amount of daily events logged into the billions. Extrapolating this out over a year can generate trillions and trillions of logged events. which can present challenges for an ad server trying to return an ad within milliseconds.
In order to store and process this much data in an economically feasible way, most of it must be rolled up and aggregated at an hourly and eventually daily level. Data at the hourly or daily level is still extremely valuable, but some of the trends and insights are not as apparent once it begins to be aggregated. Usually the raw log files are still stored, but they are not available in real time for the ad server to utilize and can only be accessed via custom reports or manual retrieval.
Fortunately technology and data storage have been advancing so what can be stored and processed today was not possible five years ago, but there are still limits. This explosion in data has created a new specialty field for big data engineers and data scientists who write algorithms to extract trends from data while processing and analyzing it as quickly as possible.
Improved ad targeting has increased the conversion rate and performance of advertisements, but it has ushered in new challenges as more and more data is required to be processed. Advancements in technology are helping, but there are still many decisions and hurdles companies must solve in order to log, analyze and return an ad in a timely manner. Fortunately, as the mobile industry matures this process will continue to get faster and faster which will benefit both advertisers and publishers.