  The past few years have produced a series of spectacular advances in the
decoding of speech from brain activity. The engine of these advances has been
the acquisition of labelled data, with increasingly large datasets acquired
from single subjects. However, participants exhibit anatomical and other
individual differences, and datasets use varied scanners and task designs. As a
result, prior work has struggled to leverage data from multiple subjects,
multiple datasets, multiple tasks, and unlabelled datasets. In turn, the field
has not benefited from the rapidly growing number of open neural data
repositories to exploit large-scale data and deep learning. To address this, we
develop an initial set of neuroscience-inspired self-supervised objectives,
together with a neural architecture, for representation learning from
heterogeneous and unlabelled neural recordings. Experimental results show that
representations learned with these objectives scale with data, generalise
across subjects, datasets, and tasks, and are also learned faster than using
only labelled data. In addition, we set new benchmarks for two foundational
speech decoding tasks. Taken together, these methods now unlock the potential
for training speech decoding models with orders of magnitude more existing
data.
