  Multilingual self-supervised learning (SSL) has often lagged behind
state-of-the-art (SOTA) methods due to the expenses and complexity required to
handle many languages. This further harms the reproducibility of SSL, which is
already limited to few research groups due to its resource usage. We show that
more powerful techniques can actually lead to more efficient pre-training,
opening SSL to more research groups. We propose WavLabLM, which extends WavLM's
joint prediction and denoising to 40k hours of data across 136 languages. To
build WavLabLM, we devise a novel multi-stage pre-training method, designed to
address the language imbalance of multilingual data. WavLabLM achieves
comparable performance to XLS-R on ML-SUPERB with less than 10% of the training
data, making SSL realizable with academic compute. We show that further
efficiency can be achieved with a vanilla HuBERT Base model, which can maintain
94% of XLS-R's performance with only 3% of the data, 4 GPUs, and limited
trials. We open-source all code and models in ESPnet.
