  Self-supervised learning (SSL) models usually require weeks of pre-training
with dozens of high-end GPUs. These models typically have a multi-headed
self-attention (MHSA) context encoder. However, MHSA takes quadratic time and
space in the input length, contributing to the high pre-training cost.
Linear-complexity alternatives to MHSA have been proposed. For instance, in
supervised training, the SummaryMixing model is the first to outperform MHSA
across multiple speech processing tasks. However, these cheaper alternatives
have not been explored for SSL yet. This paper studies a linear-complexity
context encoder for SSL for the first time. With better or equivalent
performance for the downstream tasks of the MP3S benchmark, SummaryMixing
reduces the pre-training time and peak VRAM of wav2vec 2.0 model by 18% and by
23%, respectively, leading to the pre-training of a 155M wav2vec 2.0 model
finished within one week with 4 Tesla A100 GPUs. Code is available at
https://github.com/SamsungLabs/SummaryMixing.
