  Decoding speech from brain activity is a long-awaited goal in both healthcare
and neuroscience. Invasive devices have recently led to major milestones in
that regard: deep learning algorithms trained on intracranial recordings now
start to decode elementary linguistic features (e.g. letters, words,
spectrograms). However, extending this approach to natural speech and
non-invasive brain recordings remains a major challenge. Here, we introduce a
model trained with contrastive-learning to decode self-supervised
representations of perceived speech from the non-invasive recordings of a large
cohort of healthy individuals. To evaluate this approach, we curate and
integrate four public datasets, encompassing 175 volunteers recorded with
magneto- or electro-encephalography (M/EEG), while they listened to short
stories and isolated sentences. The results show that our model can identify,
from 3 seconds of MEG signals, the corresponding speech segment with up to 41%
accuracy out of more than 1,000 distinct possibilities on average across
participants, and more than 80% in the very best participants - a performance
that allows the decoding of words and phrases absent from the training set. The
comparison of our model to a variety of baselines highlights the importance of
(i) a contrastive objective, (ii) pretrained representations of speech and
(iii) a common convolutional architecture simultaneously trained across
multiple participants. Finally, the analysis of the decoder's predictions
suggests that they primarily depend on lexical and contextual semantic
representations. Overall, this effective decoding of perceived speech from
non-invasive recordings delineates a promising path to decode language from
brain activity, without putting patients at risk for brain surgery.
