  In this paper, we investigate domain adaptation for low-resource Automatic
Speech Recognition (ASR) of target-domain data, when a well-trained ASR model
trained with a large dataset is available. We argue that in the encoder-decoder
framework, the decoder of the well-trained ASR model is largely tuned towards
the source-domain, hurting the performance of target-domain models in vanilla
transfer-learning. On the other hand, the encoder layers of the well-trained
ASR model mostly capture the acoustic characteristics. We, therefore, propose
to use the embeddings tapped from these encoder layers as features for a
downstream Conformer target-domain model and show that they provide significant
improvements. We do ablation studies on which encoder layer is optimal to tap
the embeddings, as well as the effect of freezing or updating the well-trained
ASR model's encoder layers. We further show that applying Spectral Augmentation
(SpecAug) on the proposed features (this is in addition to default SpecAug on
input spectral features) provides a further improvement on the target-domain
performance. For the LibriSpeech-100-clean data as target-domain and SPGI-5000
as a well-trained model, we get 30% relative improvement over baseline.
Similarly, with WSJ data as target-domain and LibriSpeech-960 as a well-trained
model, we get 50% relative improvement over baseline.
