  When a predictive model is in production, it must be monitored in real-time
to ensure that its performance does not suffer due to drift or abrupt changes
to data. Ideally, this is done long before learning that the performance of the
model itself has dropped by monitoring outcome data. In this paper we consider
the problem of monitoring a predictive model that identifies the need for
palliative care currently in production at the Mayo Clinic in Rochester, MN. We
introduce a framework, called \textit{Bayes Watch}, for detecting change-points
in high-dimensional longitudinal data with mixed variable types and missing
values and for determining in which variables the change-point occurred. Bayes
Watch fits an array of Gaussian Graphical Mixture Models to groupings of
homogeneous data in time, called regimes, which are modeled as the observed
states of a Markov process with unknown transition probabilities. In doing so,
Bayes Watch defines a posterior distribution on a vector of regime assignments,
which gives meaningful expressions on the probability of every possible
change-point. Bayes Watch also allows for an effective and efficient fault
detection system that assesses what features in the data where the most
responsible for a given change-point.
