  The desire to train complex machine learning algorithms and to increase the
statistical power in association studies drives neuroimaging research to use
ever-larger datasets. The most obvious way to increase sample size is by
pooling scans from independent studies. However, simple pooling is often
ill-advised as selection, measurement, and confounding biases may creep in and
yield spurious correlations. In this work, we combine 35,320 magnetic resonance
images of the brain from 17 studies to examine bias in neuroimaging. In the
first experiment, Name That Dataset, we provide empirical evidence for the
presence of bias by showing that scans can be correctly assigned to their
respective dataset with 71.5% accuracy. Given such evidence, we take a closer
look at confounding bias, which is often viewed as the main shortcoming in
observational studies. In practice, we neither know all potential confounders
nor do we have data on them. Hence, we model confounders as unknown, latent
variables. Kolmogorov complexity is then used to decide whether the confounded
or the causal model provides the simplest factorization of the graphical model.
Finally, we present methods for dataset harmonization and study their ability
to remove bias in imaging features. In particular, we propose an extension of
the recently introduced ComBat algorithm to control for global variation across
image features, inspired by adjusting for population stratification in
genetics. Our results demonstrate that harmonization can reduce
dataset-specific information in image features. Further, confounding bias can
be reduced and even turned into a causal relationship. However, harmonziation
also requires caution as it can easily remove relevant subject-specific
information. Code is available at https://github.com/ai-med/Dataset-Bias.
