  Large pre-trained language models (PLMs) have shown remarkable performance
across various natural language understanding (NLU) tasks, particularly in
low-resource settings. Nevertheless, their potential in Automatic Speech
Recognition (ASR) remains largely unexplored. This study investigates the
potential usage of PLMs for language modelling in ASR. We compare the
application of large-scale text sampling and probability conversion for
approximating GPT-2 into an n-gram model. Furthermore, we introduce a
vocabulary-restricted decoding method for random sampling, and evaluate the
effects of domain difficulty and data size on the usability of generated text.
Our findings across eight domain-specific corpora support the use of
sampling-based approximation and show that interpolating with a large sampled
corpus improves test perplexity over a baseline trigram by 15%. Our
vocabulary-restricted decoding method pushes this improvement further by 5% in
domain-specific settings.
