  ASR model deployment environment is ever-changing, and the incoming speech
can be switched across different domains during a session. This brings a
challenge for effective domain adaptation when only target domain text data is
available, and our objective is to obtain obviously improved performance on the
target domain while the performance on the general domain is less undermined.
In this paper, we propose an adaptive LM fusion approach called internal
language model estimation based adaptive domain adaptation (ILME-ADA). To
realize such an ILME-ADA, an interpolated log-likelihood score is calculated
based on the maximum of the scores from the internal LM and the external LM
(ELM) respectively. We demonstrate the efficacy of the proposed ILME-ADA method
with both RNN-T and LAS modeling frameworks employing neural network and n-gram
LMs as ELMs respectively on two domain specific (target) test sets. The
proposed method can achieve significantly better performance on the target test
sets while it gets minimal performance degradation on the general test set,
compared with both shallow and ILME-based LM fusion methods.
