  Most end-to-end (E2E) speech recognition models are composed of encoder and
decoder blocks that perform acoustic and language modeling functions.
Pretrained large language models (LLMs) have the potential to improve the
performance of E2E ASR. However, integrating a pretrained language model into
an E2E speech recognition model has shown limited benefits due to the
mismatches between text-based LLMs and those used in E2E ASR. In this paper, we
explore an alternative approach by adapting a pretrained LLMs to speech. Our
experiments on fully-formatted E2E ASR transcription tasks across various
domains demonstrate that our approach can effectively leverage the strengths of
pretrained LLMs to produce more readable ASR transcriptions. Our model, which
is based on the pretrained large language models with either an encoder-decoder
or decoder-only structure, surpasses strong ASR models such as Whisper, in
terms of recognition error rate, considering formats like punctuation and
capitalization as well.
