  Self-supervised learning (SSL) techniques have achieved remarkable results in
various speech processing tasks. Nonetheless, a significant challenge remains
in reducing the reliance on vast amounts of speech data for pre-training. This
paper proposes to address this challenge by leveraging synthetic speech to
augment a low-resource pre-training corpus. We construct a high-quality
text-to-speech (TTS) system with limited resources using SSL features and
generate a large synthetic corpus for pre-training. Experimental results
demonstrate that our proposed approach effectively reduces the demand for
speech data by 90% with only slight performance degradation. To the best of our
knowledge, this is the first work aiming to enhance low-resource
self-supervised learning in speech processing.
