  Deep learning is leading to major advances in the realm of brain decoding
from functional Magnetic Resonance Imaging (fMRI). However, the large
inter-subject variability in brain characteristics has limited most studies to
train models on one subject at a time. Consequently, this approach hampers the
training of deep learning models, which typically requires very large datasets.
Here, we propose to boost brain decoding by aligning brain responses to videos
and static images across subjects. Compared to the anatomically-aligned
baseline, our method improves out-of-subject decoding performance by up to 75%.
Moreover, it also outperforms classical single-subject approaches when fewer
than 100 minutes of data is available for the tested subject. Furthermore, we
propose a new multi-subject alignment method, which obtains comparable results
to that of classical single-subject approaches while improving out-of-subject
generalization. Finally, we show that this method aligns neural representations
in accordance with brain anatomy. Overall, this study lays the foundations for
leveraging extensive neuroimaging datasets and enhancing the decoding of
individuals with a limited amount of brain recordings.
