  Recent studies have shown that using an external Language Model (LM) benefits
the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens
that appear less frequently in the training set is still quite challenging. The
long-tail prediction problems have been widely studied in many applications,
but only been addressed by a few studies for ASR and LMs. In this paper, we
propose a new memory augmented lookup dictionary based Transformer architecture
for LM. The newly introduced lookup dictionary incorporates rich contextual
information in training set, which is vital to correctly predict long-tail
tokens. With intensive experiments on Chinese and English data sets, our
proposed method is proved to outperform the baseline Transformer LM by a great
margin on both word/character error rate and tail tokens error rate. This is
achieved without impact on the decoding efficiency. Overall, we demonstrate the
effectiveness of our proposed method in boosting the ASR decoding performance,
especially for long-tail tokens.
