  This paper addresses the problem of improving speech recognition accuracy
with lattice rescoring in low-resource languages where the baseline language
model is insufficient for generating inclusive lattices. We minimally augment
the baseline language model with word unigram counts that are present in a
larger text corpus of the target language but absent in the baseline. The
lattices generated after decoding with such an augmented baseline language
model are more comprehensive. We obtain 21.8% (Telugu) and 41.8% (Kannada)
relative word error reduction with our proposed method. This reduction in word
error rate is comparable to 21.5% (Telugu) and 45.9% (Kannada) relative word
error reduction obtained by decoding with full Wikipedia text augmented
language mode while our approach consumes only 1/8th the memory. We demonstrate
that our method is comparable with various text selection-based language model
augmentation and also consistent for data sets of different sizes. Our approach
is applicable for training speech recognition systems under low resource
conditions where speech data and compute resources are insufficient, while
there is a large text corpus that is available in the target language. Our
research involves addressing the issue of out-of-vocabulary words of the
baseline in general and does not focus on resolving the absence of named
entities. Our proposed method is simple and yet computationally less expensive.
