  Self-supervised learning techniques are celebrating immense success in
natural language processing (NLP) by enabling models to learn from broad
language data at unprecedented scales. Here, we aim to leverage the success of
these techniques for mental state decoding, where researchers aim to identify
specific mental states (e.g., the experience of anger or joy) from brain
activity. To this end, we devise a set of novel self-supervised learning
frameworks for neuroimaging data inspired by prominent learning frameworks in
NLP. At their core, these frameworks learn the dynamics of brain activity by
modeling sequences of activity akin to how sequences of text are modeled in
NLP. We evaluate the frameworks by pre-training models on a broad neuroimaging
dataset spanning functional Magnetic Resonance Imaging data from 11,980
experimental runs of 1,726 individuals across 34 datasets, and subsequently
adapting the pre-trained models to benchmark mental state decoding datasets.
The pre-trained models transfer well, generally outperforming baseline models
trained from scratch, while models trained in a learning framework based on
causal language modeling clearly outperform the others.
