  Speech and text are two major forms of human language. The research community
has been focusing on mapping speech to text or vice versa for many years.
However, in the field of language modeling, very little effort has been made to
model them jointly. In light of this, we explore joint language modeling for
speech units and text. Specifically, we compare different speech tokenizers to
transform continuous speech signals into discrete units and use different
methods to construct mixed speech-text data. We introduce automatic metrics to
evaluate how well the joint LM mixes speech and text. We also fine-tune the LM
on downstream spoken language understanding (SLU) tasks with different
modalities (speech or text) and test its performance to assess the model's
learning of shared representations. Our results show that by mixing speech
units and text with our proposed mixing techniques, the joint LM improves over
a speech-only baseline on SLU tasks and shows zero-shot cross-modal
transferability.
