  Deep learning associated with neurological signals is poised to drive major
advancements in diverse fields such as medical diagnostics,
neurorehabilitation, and brain-computer interfaces. The challenge in harnessing
the full potential of these signals lies in the dependency on extensive,
high-quality annotated data, which is often scarce and expensive to acquire,
requiring specialized infrastructure and domain expertise. To address the
appetite for data in deep learning, we present Neuro-BERT, a self-supervised
pre-training framework of neurological signals based on masked autoencoding in
the Fourier domain. The intuition behind our approach is simple: frequency and
phase distribution of neurological signals can reveal intricate neurological
activities. We propose a novel pre-training task dubbed Fourier Inversion
Prediction (FIP), which randomly masks out a portion of the input signal and
then predicts the missing information using the Fourier inversion theorem.
Pre-trained models can be potentially used for various downstream tasks such as
sleep stage classification and gesture recognition. Unlike contrastive-based
methods, which strongly rely on carefully hand-crafted augmentations and
siamese structure, our approach works reasonably well with a simple transformer
encoder with no augmentation requirements. By evaluating our method on several
benchmark datasets, we show that Neuro-BERT improves downstream
neurological-related tasks by a large margin.
