  Self-supervised learning methods such as wav2vec 2.0 have shown promising
results in learning speech representations from unlabelled and untranscribed
speech data that are useful for speech recognition. Since these representations
are learned without any task-specific supervision, they can also be useful for
other voice-activated tasks like speaker verification, keyword spotting,
emotion classification etc. In our work, we propose a general purpose framework
for adapting a pre-trained wav2vec 2.0 model for different voice-activated
tasks. We develop downstream network architectures that operate on the
contextualized speech representations of wav2vec 2.0 to adapt the
representations for solving a given task. Finally, we extend our framework to
perform multi-task learning by jointly optimizing the network parameters on
multiple voice activated tasks using a shared transformer backbone. Both of our
single and multi-task frameworks achieve state-of-the-art results in speaker
verification and keyword spotting benchmarks. Our best performing models
achieve 1.98% and 3.15% EER on VoxCeleb1 test set when trained on VoxCeleb2 and
VoxCeleb1 respectively, and 98.23% accuracy on Google Speech Commands v1.0
keyword spotting dataset.
