  Integrating external language models (LMs) into end-to-end (E2E) models
remains a challenging task for domain-adaptive speech recognition. Recently,
internal language model estimation (ILME)-based LM fusion has shown significant
word error rate (WER) reduction from Shallow Fusion by subtracting a weighted
internal LM score from an interpolation of E2E model and external LM scores
during beam search. However, on different test sets, the optimal LM
interpolation weights vary over a wide range and have to be tuned extensively
on well-matched validation sets. In this work, we perform LM fusion in the
minimum WER (MWER) training of an E2E model to obviate the need for LM weights
tuning during inference. Besides MWER training with Shallow Fusion (MWER-SF),
we propose a novel MWER training with ILME (MWER-ILME) where the ILME-based
fusion is conducted to generate N-best hypotheses and their posteriors.
Additional gradient is induced when internal LM is engaged in MWER-ILME loss
computation. During inference, LM weights pre-determined in MWER training
enable robust LM integrations on test sets from different domains. Experimented
with 30K-hour trained transformer transducers, MWER-ILME achieves on average
8.8% and 5.8% relative WER reductions from MWER and MWER-SF training,
respectively, on 6 different test sets
