  This paper proposes a way of protecting probabilistic prediction models
against changes in the data distribution, concentrating on the case of
classification and paying particular attention to binary classification. This
is important in applications of machine learning, where the quality of a
trained prediction algorithm may drop significantly in the process of its
exploitation. Our techniques are based on recent work on conformal test
martingales and older work on prediction with expert advice, namely tracking
the best expert.
