  Changes in the data distribution at test time can have deleterious effects on
the performance of predictive models $p(y|x)$. We consider situations where
there are additional meta-data labels (such as group labels), denoted by $z$,
that can account for such changes in the distribution. In particular, we assume
that the prior distribution $p(y, z)$, which models the dependence between the
class label $y$ and the "nuisance" factors $z$, may change across domains,
either due to a change in the correlation between these terms, or a change in
one of their marginals. However, we assume that the generative model for
features $p(x|y,z)$ is invariant across domains. We note that this corresponds
to an expanded version of the widely used "label shift" assumption, where the
labels now also include the nuisance factors $z$. Based on this observation, we
propose a test-time label shift correction that adapts to changes in the joint
distribution $p(y, z)$ using EM applied to unlabeled samples from the target
domain distribution, $p_t(x)$. Importantly, we are able to avoid fitting a
generative model $p(x|y, z)$, and merely need to reweight the outputs of a
discriminative model $p_s(y, z|x)$ trained on the source distribution. We
evaluate our method, which we call "Test-Time Label-Shift Adaptation" (TTLSA),
on several standard image and text datasets, as well as the CheXpert chest
X-ray dataset, and show that it improves performance over methods that target
invariance to changes in the distribution, as well as baseline empirical risk
minimization methods. Code for reproducing experiments is available at
https://github.com/nalzok/test-time-label-shift .
