  We present Spectron, a novel approach to adapting pre-trained large language
models (LLMs) to perform spoken question answering (QA) and speech
continuation. By endowing the LLM with a pre-trained speech encoder, our model
becomes able to take speech inputs and generate speech outputs. The entire
system is trained end-to-end and operates directly on spectrograms, simplifying
our architecture. Key to our approach is a training objective that jointly
supervises speech recognition, text continuation, and speech synthesis using
only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within
a single decoding pass. Our method surpasses existing spoken language models in
speaker preservation and semantic coherence. Furthermore, the proposed model
improves upon direct initialization in retaining the knowledge of the original
LLM as demonstrated through spoken QA datasets. We release our audio samples
(https://michelleramanovich.github.io/spectron/spectron) and spoken QA dataset
(https://github.com/google-research-datasets/LLAMA1-Test-Set).
