  It was shown in literature that speech representations extracted by
self-supervised pre-trained models exhibit similarities with brain activations
of human for speech perception and fine-tuning speech representation models on
downstream tasks can further improve the similarity. However, it still remains
unclear if this similarity can be used to optimize the pre-trained speech
models. In this work, we therefore propose to use the brain activations
recorded by fMRI to refine the often-used wav2vec2.0 model by aligning model
representations toward human neural responses. Experimental results on SUPERB
reveal that this operation is beneficial for several downstream tasks, e.g.,
speaker verification, automatic speech recognition, intent classification.One
can then consider the proposed method as a new alternative to improve
self-supervised speech models.
