  We investigate the effectiveness of using a large ensemble of advanced neural
language models (NLMs) for lattice rescoring on automatic speech recognition
(ASR) hypotheses. Previous studies have reported the effectiveness of combining
a small number of NLMs. In contrast, in this study, we combine up to eight
NLMs, i.e., forward/backward long short-term memory/Transformer-LMs that are
trained with two different random initialization seeds. We combine these NLMs
through iterative lattice generation. Since these NLMs work complementarily
with each other, by combining them one by one at each rescoring iteration,
language scores attached to given lattice arcs can be gradually refined.
Consequently, errors of the ASR hypotheses can be gradually reduced. We also
investigate the effectiveness of carrying over contextual information (previous
rescoring results) across a lattice sequence of a long speech such as a lecture
speech. In experiments using a lecture speech corpus, by combining the eight
NLMs and using context carry-over, we obtained a 24.4% relative word error rate
reduction from the ASR 1-best baseline. For further comparison, we performed
simultaneous (i.e., non-iterative) NLM combination and 100-best rescoring using
the large ensemble of NLMs, which confirmed the advantage of lattice rescoring
with iterative NLM combination.
