  Generative Spoken Language Modeling research focuses on optimizing speech
Language Models (LMs) using raw audio recordings without accessing any textual
supervision. Such speech LMs usually operate over discrete units obtained from
quantizing internal representations of self-supervised models. Although such
units show impressive modeling results, their robustness capabilities have not
been extensively investigated. This work focuses on improving the robustness of
discrete input representations for generative spoken language modeling. First,
we formally define how to measure the robustness of such representations to
various signal variations that do not alter the spoken information (e.g.,
time-stretch). Next, we empirically demonstrate how current state-of-the-art
representation models lack robustness to such variations. To overcome this, we
propose an effective and efficient method to learn robust discrete speech
representation for generative spoken language modeling. The proposed approach
is based on applying a set of signal transformations to the speech signal and
optimizing the model using an iterative pseudo-labeling scheme. Our method
significantly improves over the evaluated baselines when considering encoding
and modeling metrics. We additionally evaluate our method on the
speech-to-speech translation task, considering Spanish-English and
French-English translations, and show the proposed approach outperforms the
evaluated baselines.
