  Speech Language Models (SLMs) aim to learn language from raw audio, without
textual resources. Despite significant advances, our current models exhibit
weak syntax and semantic abilities. However, if the scaling properties of
neural language models hold for the speech modality, these abilities will
improve as the amount of compute used for training increases. In this paper, we
use models of this scaling behavior to estimate the scale at which our current
methods will yield a SLM with the English proficiency of text-based Large
Language Models (LLMs). We establish a strong correlation between pre-training
loss and downstream syntactic and semantic performance in SLMs and LLMs, which
results in predictable scaling of linguistic performance. We show that the
linguistic performance of SLMs scales up to three orders of magnitude more
slowly than that of text-based LLMs. Additionally, we study the benefits of
synthetic data designed to boost semantic understanding and the effects of
coarser speech tokenization.
