  Domain shift, the mismatch between training and testing data characteristics,
causes significant degradation in the predictive performance in multi-source
imaging scenarios. In medical imaging, the heterogeneity of population,
scanners and acquisition protocols at different sites presents a significant
domain shift challenge and has limited the widespread clinical adoption of
machine learning models. Harmonization methods which aim to learn a
representation of data invariant to these differences are the prevalent tools
to address domain shift, but they typically result in degradation of predictive
accuracy. This paper takes a different perspective of the problem: we embrace
this disharmony in data and design a simple but effective framework for
tackling domain shift. The key idea, based on our theoretical arguments, is to
build a pretrained classifier on the source data and adapt this model to new
data. The classifier can be fine-tuned for intra-site domain adaptation. We can
also tackle situations where we do not have access to ground-truth labels on
target data; we show how one can use auxiliary tasks for adaptation; these
tasks employ covariates such as age, gender and race which are easy to obtain
but nevertheless correlated to the main task. We demonstrate substantial
improvements in both intra-site domain adaptation and inter-site domain
generalization on large-scale real-world 3D brain MRI datasets for classifying
Alzheimer's disease and schizophrenia.
