  Self-supervised learning (SSL) has shown tremendous success in various
speech-related downstream tasks, including Automatic Speech Recognition (ASR).
The output embeddings of the SSL model are treated as powerful short-time
representations of the speech signal. However, in the ASR task, the main
objective is to get the correct sequence of acoustic units, characters, or
byte-pair encodings (BPEs). Usually, encoder-decoder architecture works
exceptionally well for a sequence-to-sequence task like ASR. Therefore, in this
paper, we propose a new paradigm that exploits the power of a decoder during
self-supervised learning. We use Hidden Unit BERT (HuBERT) SSL framework to
compute the conventional masked prediction loss for the encoder. In addition,
we have introduced a decoder in the SSL framework and proposed a target
preparation strategy for the decoder. Finally, we use a multitask SSL setup
wherein we jointly optimize both the encoder and decoder losses. We hypothesize
that the presence of a decoder in the SSL model helps it learn an acoustic
unit-based language model, which might improve the performance of an ASR
downstream task. We compare our proposed SSL model with HuBERT and show up to
25% relative improvement in performance on ASR by finetuning on various
LibriSpeech subsets.
