  Operations is a key challenge in the domain of machine learning pipeline
deployments involving monitoring and management of real-time prediction
quality. Typically, metrics like accuracy, RMSE etc., are used to track the
performance of models in deployment. However, these metrics cannot be
calculated in production due to the absence of labels. We propose using an ML
algorithm, Model Performance Predictor (MPP), to track the performance of the
models in deployment. We argue that an ensemble of such metrics can be used to
create a score representing the prediction quality in production. This in turn
facilitates formulation and customization of ML alerts, that can be escalated
by an operations team to the data science team. Such a score automates
monitoring and enables ML deployments at scale.
