  An end-to-end (E2E) ASR model implicitly learns a prior Internal Language
Model (ILM) from the training transcripts. To fuse an external LM using Bayes
posterior theory, the log likelihood produced by the ILM has to be accurately
estimated and subtracted. In this paper we propose two novel approaches to
estimate the ILM based on Listen-Attend-Spell (LAS) framework. The first method
is to replace the context vector of the LAS decoder at every time step with a
vector that is learned with training transcripts. Furthermore, we propose
another method that uses a lightweight feed-forward network to directly map
query vector to context vector in a dynamic sense. Since the context vectors
are learned by minimizing the perplexities on training transcripts, and their
estimation is independent of encoder output, hence the ILMs are accurately
learned for both methods. Experiments show that the ILMs achieve the lowest
perplexity, indicating the efficacy of the proposed methods. In addition, they
also significantly outperform the shallow fusion method, as well as two
previously proposed ILM Estimation (ILME) approaches on several datasets.
