  In this paper, we use spectral analysis to investigate transfer learning and
study model sensitivity to frequency shortcuts in medical imaging. By analyzing
the power spectrum density of both pre-trained and fine-tuned model gradients,
as well as artificially generated frequency shortcuts, we observe notable
differences in learning priorities between models pre-trained on natural vs
medical images, which generally persist during fine-tuning. We find that when a
model's learning priority aligns with the power spectrum density of an
artifact, it results in overfitting to that artifact. Based on these
observations, we show that source data editing can alter the model's resistance
to shortcut learning.
