  Utilizing text-only data with an external language model (ELM) in end-to-end
RNN-Transducer (RNN-T) for speech recognition is challenging. Recently, a class
of methods such as density ratio (DR) and internal language model estimation
(ILME) have been developed, outperforming the classic shallow fusion (SF)
method. The basic idea behind these methods is that RNN-T posterior should
first subtract the implicitly learned internal language model (ILM) prior, in
order to integrate the ELM. While recent studies suggest that RNN-T only learns
some low-order language model information, the DR method uses a well-trained
neural language model with full context, which may be inappropriate for the
estimation of ILM and deteriorate the integration performance. Based on the DR
method, we propose a low-order density ratio method (LODR) by replacing the
estimation with a low-order weak language model. Extensive empirical
experiments are conducted on both in-domain and cross-domain scenarios on
English LibriSpeech & Tedlium-2 and Chinese WenetSpeech & AISHELL-1 datasets.
It is shown that LODR consistently outperforms SF in all tasks, while
performing generally close to ILME and better than DR in most tests.
