  Classifier predictions often rely on the assumption that new observations
come from the same distribution as training data. When the underlying
distribution changes, so does the optimal classification rule, and performance
may degrade. We consider the problem of detecting such a change in distribution
in sequentially-observed, unlabeled classification data. We focus on label
shift changes to the distribution, where the class priors shift but the class
conditional distributions remain unchanged. We reduce this problem to the
problem of detecting a change in the one-dimensional classifier scores, leading
to simple nonparametric sequential changepoint detection procedures. Our
procedures leverage classifier training data to estimate the detection
statistic, and converge to their parametric counterparts in the size of the
training data. In simulations, we show that our method outperforms other
detection procedures in this label shift setting.
