  It takes skill to build a meaningful predictive model even with the abundance
of implementations of modern machine learning algorithms and readily available
computing resources. Building a model becomes challenging if hundreds of
terabytes of data need to be processed to produce the training data set. In a
digital advertising technology setting, we are faced with the need to build
thousands of such models that predict user behavior and power advertising
campaigns in a 24/7 chaotic real-time production environment. As data
scientists, we also have to convince other internal departments critical to
implementation success, our management, and our customers that our machine
learning system works. In this paper, we present the details of the design and
implementation of an automated, robust machine learning platform that impacts
billions of advertising impressions monthly. This platform enables us to
continuously optimize thousands of campaigns over hundreds of millions of
users, on multiple continents, against varying performance objectives.
