  We present a method for transferring pre-trained self-supervised (SSL) speech
representations to multiple languages. There is an abundance of unannotated
speech, so creating self-supervised representations from raw audio and
fine-tuning on small annotated datasets is a promising direction to build
speech recognition systems. SSL models generally perform SSL on raw audio in a
pre-training phase and then fine-tune on a small fraction of annotated data.
Such models have produced state of the art results for ASR. However, these
models are very expensive to pre-train. We use an existing wav2vec 2.0 model
and tackle the problem of learning new language representations while utilizing
existing model knowledge. Crucially we do so without catastrophic forgetting of
the existing language representation. We use adapter modules to speed up
pre-training a new language task. Our model can decrease pre-training times by
32% when learning a new language task, and learn this new audio-language
representation without forgetting previous language representation. We evaluate
by applying these language representations to automatic speech recognition.
