  Class distribution plays an important role in learning deep classifiers. When
the proportion of each class in the test set differs from the training set, the
performance of classification nets usually degrades. Such a label distribution
shift problem is common in medical diagnosis since the prevalence of disease
vary over location and time. In this paper, we propose the first method to
tackle label shift for medical image classification, which effectively adapt
the model learned from a single training label distribution to arbitrary
unknown test label distribution. Our approach innovates distribution
calibration to learn multiple representative classifiers, which are capable of
handling different one-dominating-class distributions. When given a test image,
the diverse classifiers are dynamically aggregated via the consistency-driven
test-time adaptation, to deal with the unknown test label distribution. We
validate our method on two important medical image classification tasks
including liver fibrosis staging and COVID-19 severity prediction. Our
experiments clearly show the decreased model performance under label shift.
With our method, model performance significantly improves on all the test
datasets with different label shifts for both medical image diagnosis tasks.
