  In user targeting automation systems, concept drift in input data is one of
the main challenges. It deteriorates model performance on new data over time.
Previous research on concept drift mostly proposed model retraining after
observing performance decreases. However, this approach is suboptimal because
the system fixes the problem only after suffering from poor performance on new
data. Here, we introduce an adversarial validation approach to concept drift
problems in user targeting automation systems. With our approach, the system
detects concept drift in new data before making inference, trains a model, and
produces predictions adapted to the new data. We show that our approach
addresses concept drift effectively with the AutoML3 Lifelong Machine Learning
challenge data as well as in Uber's internal user targeting automation system,
MaLTA.
