  When monitoring machine learning systems, two-sample tests of homogeneity
form the foundation upon which existing approaches to drift detection build.
They are used to test for evidence that the distribution underlying recent
deployment data differs from that underlying the historical reference data.
Often, however, various factors such as time-induced correlation mean that
batches of recent deployment data are not expected to form an i.i.d. sample
from the historical data distribution. Instead we may wish to test for
differences in the distributions conditional on \textit{context} that is
permitted to change. To facilitate this we borrow machinery from the causal
inference domain to develop a more general drift detection framework built upon
a foundation of two-sample tests for conditional distributional treatment
effects. We recommend a particular instantiation of the framework based on
maximum conditional mean discrepancies. We then provide an empirical study
demonstrating its effectiveness for various drift detection problems of
practical interest, such as detecting drift in the distributions underlying
subpopulations of data in a manner that is insensitive to their respective
prevalences. The study additionally demonstrates applicability to
ImageNet-scale vision problems.
