How redesigning the Demand-Side Platform and improving its real-time bidding component helped an advertising company gather valuable information on external user engagement and reach the latency level of less than 100 ms.
*instinctools company was involved in the Real Time Bidding project that uses the Personal Retargeting technique. It is about showing ads to users, who have visited an advertiser’s site, however, failed to perform the conversion action. Personal Retargeting brings them back and shows each user a specific scenario,
depending on their previous action: viewing an item, comparing an item to other modifications etc.
DSP (Demand Side Platform) makes each 1 click cheaper. While using Trading Desk, advertisers adjust bids according to the value of their target audience, as well as to the desired impressions and clicks volume. The platform compares the data with the information on traders and bids, buying impressions as cheap as
possible and selecting the best matching audience. The whole process takes a split second: while the page is still loading.
The existing platform has been maintained and augmented with the Demand Side Platform, a system that organizes auctions for advertisers, and a user data management system. The platform has been integrated with the largest SSP platforms, such as Facebook Exchange and Taboola, employing HTTP cookies to gather information on users.
We have done an analysis of existing components and put together technical manuals. Having revealed and eliminated the failure caused in the course of maintenance, we achieved stable system’s operation.
We also have:
The system design was based on wellknown Amazon cloud technologies like Elastic Compute Cloud, Elastic Load Balancer, Simple Storage Cloud, Elastic MapReduce, DynamoDB, etc.
Aggregated reporting included processing of huge data amounts received from SSP and DMP components. It was implemented using Hadoop jobs executed in Amazon EMR. The results were aggregated in Amazon Redshift databases.
As a part of asynchronous operation processing, the several messaging platforms were used. NSQ provided distributed queues for high amount data processing. Sidekiq was used to implement business logic messages processing through the set of components written in Ruby.
Application metrics were processed using Graphite and were shown by Grafana.
Continuous integration processes were implemented based on Gitflow by using Github + Travis services.
For the needs of infrastructure hosting, data storage, data sharing the following services were used: Amazon EC2, Amazon ELB, Amazon S3.
The components were developed in modern programming languages. Go language capabilities gave an opportunity to implement performance critical items. Scripting functionality which included component integration, configuration and deployment were completely implemented with Ruby and related technologies.
Multiple database management systems were used based on the required functionality: DynamoDB and Aerospike used to collect huge amount of data related to outside user activity, Redis databases satisfied the needs of temporary data keeping, PostgreSQL and Amazon Redshift provided strong consistency storage and processing of data related business activity and reporting.
Rollbar service was used to track errors occurred in a production environment.
The client has obtained a stable version of the platform together with the build products, as well as an option to solve integration issues quickly and at a low cost in the future.
The augmented platform benefits are:
The client was pleased with our fulfillment of the requirements and the recommendations of advertising partners. We are currently continuing to maintain the stable operation of the platform.
Go language
NSQ
DynamoDB
Aerospike
Redis
Ruby
Sidekiq
PostgreSQL
GitHub
Rollbar
Gitflow
Grafana
Graphite
Hadoop jobs
Travis
Elastic Compute Cloud
Elastic Load Balancer
Simple Storage Cloud
Elastic MapReduce
DynamoDB
Amazon EC2
Amazon ELB
Amazon Redshift
Amazon EMR
Amazon S3
