  Deployment of machine learning (ML) algorithms in production for extended
periods of time has uncovered new challenges such as monitoring and management
of real-time prediction quality of a model in the absence of labels. However,
such tracking is imperative to prevent catastrophic business outcomes resulting
from incorrect predictions. The scale of these deployments makes manual
monitoring prohibitive, making automated techniques to track and raise alerts
imperative. We present a framework, ML Health, for tracking potential drops in
the predictive performance of ML models in the absence of labels. The framework
employs diagnostic methods to generate alerts for further investigation. We
develop one such method to monitor potential problems when production data
patterns do not match training data distributions. We demonstrate that our
method performs better than standard "distance metrics", such as RMSE,
KL-Divergence, and Wasserstein at detecting issues with mismatched data sets.
Finally, we present a working system that incorporates the ML Health approach
to monitor and manage ML deployments within a realistic full production ML
lifecycle.
