  Speech discrete representation has proven effective in various downstream
applications due to its superior compression rate of the waveform, fast
convergence during training, and compatibility with other modalities. Discrete
units extracted from self-supervised learning (SSL) models have emerged as a
prominent approach for obtaining speech discrete representation. However, while
discrete units have shown effectiveness compared to spectral features, they
still lag behind continuous SSL representations. In this work, we propose MMM,
a multi-layer multi-residual multi-stream discrete units extraction method from
SSL. Specifically, we introduce iterative residual vector quantization with
K-means for different layers in an SSL model to extract multi-stream speech
discrete representation. Through extensive experiments in speech recognition,
speech resynthesis, and text-to-speech, we demonstrate the proposed MMM can
surpass or on-par with neural codec's performance under various conditions.
