  Research in many fields has shown that transfer learning (TL) is well-suited
to improve the performance of deep learning (DL) models in datasets with small
numbers of samples. This empirical success has triggered interest in the
application of TL to cognitive decoding analyses with functional neuroimaging
data. Here, we systematically evaluate TL for the application of DL models to
the decoding of cognitive states (e.g., viewing images of faces or houses) from
whole-brain functional Magnetic Resonance Imaging (fMRI) data. We first
pre-train two DL architectures on a large, public fMRI dataset and subsequently
evaluate their performance in an independent experimental task and a fully
independent dataset. The pre-trained models consistently achieve higher
decoding accuracies and generally require less training time and data than
model variants that were not pre-trained, clearly underlining the benefits of
pre-training. We demonstrate that these benefits arise from the ability of the
pre-trained models to reuse many of their learned features when training with
new data, providing deeper insights into the mechanisms giving rise to the
benefits of pre-training. Yet, we also surface nuanced challenges for
whole-brain cognitive decoding with DL models when interpreting the decoding
decisions of the pre-trained models, as these have learned to utilize the fMRI
data in unforeseen and counterintuitive ways to identify individual cognitive
states.
