  Transfer learning has become an essential part of medical imaging
classification algorithms, often leveraging ImageNet weights. However, the
domain shift from natural to medical images has prompted alternatives such as
RadImageNet, often demonstrating comparable classification performance.
However, it remains unclear whether the performance gains from transfer
learning stem from improved generalization or shortcut learning. To address
this, we investigate potential confounders -- whether synthetic or sampled from
the data -- across two publicly available chest X-ray and CT datasets. We show
that ImageNet and RadImageNet achieve comparable classification performance,
yet ImageNet is much more prone to overfitting to confounders. We recommend
that researchers using ImageNet-pretrained models reexamine their model
robustness by conducting similar experiments. Our code and experiments are
available at https://github.com/DovileDo/source-matters.
