  Unsupervised cross-lingual speech representation learning (XLSR) has recently
shown promising results in speech recognition by leveraging vast amounts of
unlabeled data across multiple languages. However, standard XLSR model suffers
from language interference problem due to the lack of language specific
modeling ability. In this work, we investigate language adaptive training on
XLSR models. More importantly, we propose a novel language adaptive
pre-training approach based on sparse sharing sub-networks. It makes room for
language specific modeling by pruning out unimportant parameters for each
language, without requiring any manually designed language specific component.
After pruning, each language only maintains a sparse sub-network, while the
sub-networks are partially shared with each other. Experimental results on a
downstream multilingual speech recognition task show that our proposed method
significantly outperforms baseline XLSR models on both high resource and low
resource languages. Besides, our proposed method consistently outperforms other
adaptation methods and requires fewer parameters.
