  The statistical machine learning community has demonstrated considerable
resourcefulness over the years in developing highly expressive tools for
estimation, prediction, and inference. The bedrock assumptions underlying these
developments are that the data comes from a fixed population and displays
little heterogeneity. But reality is significantly more complex: statistical
models now routinely fail when released into real-world systems and scientific
applications, where such assumptions rarely hold. Consequently, we pursue a
different path in this paper vis-a-vis the well-worn trail of developing new
methodology for estimation and prediction. In this paper, we develop tools and
theory for detecting and identifying regions of the covariate space
(subpopulations) where model performance has begun to degrade, and study
intervening to fix these failures through refitting. We present empirical
results with three real-world data sets -- including a time series involving
forecasting the incidence of COVID-19 -- showing that our methodology generates
interpretable results, is useful for tracking model performance, and can boost
model performance through refitting. We complement these empirical results with
theory proving that our methodology is minimax optimal for recovering anomalous
subpopulations as well as refitting to improve accuracy in a structured normal
means setting.
