  Recently, self-supervised learning (SSL) from unlabelled speech data has
gained increased attention in the automatic speech recognition (ASR) community.
Typical SSL methods include autoregressive predictive coding (APC), Wav2vec2.0,
and hidden unit BERT (HuBERT). However, SSL models are biased to the
pretraining data. When SSL models are finetuned with data from another domain,
domain shifting occurs and might cause limited knowledge transfer for
downstream tasks. In this paper, we propose a novel framework, domain
responsible adaptation and finetuning (DRAFT), to reduce domain shifting in
pretrained speech models, and evaluate it for a causal and non-causal
transformer. For the causal transformer, an extension of APC (E-APC) is
proposed to learn richer information from unlabelled data by using multiple
temporally-shifted sequences to perform prediction. For the non-causal
transformer, various solutions for using the bidirectional APC (Bi-APC) are
investigated. In addition, the DRAFT framework is examined for Wav2vec2.0 and
HuBERT methods, which use non-causal transformers as the backbone. The
experiments are conducted on child ASR (using the OGI and MyST databases) using
SSL models trained with unlabelled adult speech data from Librispeech. The
relative WER improvements of up to 19.7% on the two child tasks are observed
when compared to the pretrained models without adaptation. With the proposed
methods (E-APC and DRAFT), the relative WER improvements are even larger (30%
and 19% on the OGI and MyST data, respectively) when compared to the models
without using pretraining methods.
