  Self-supervised learning (SSL) is a powerful tool that allows learning of
underlying representations from unlabeled data. Transformer based models such
as wav2vec 2.0 and HuBERT are leading the field in the speech domain. Generally
these models are fine-tuned on a small amount of labeled data for a downstream
task such as Automatic Speech Recognition (ASR). This involves re-training the
majority of the model for each task. Adapters are small lightweight modules
which are commonly used in Natural Language Processing (NLP) to adapt
pre-trained models to new tasks. In this paper we propose applying adapters to
wav2vec 2.0 to reduce the number of parameters required for downstream ASR
tasks, and increase scalability of the model to multiple tasks or languages.
Using adapters we can perform ASR while training fewer than 10% of parameters
per task compared to full fine-tuning with little degradation of performance.
Ablations show that applying adapters into just the top few layers of the
pre-trained network gives similar performance to full transfer, supporting the
theory that higher pre-trained layers encode more phonemic information, and
further optimizing efficiency.
