  Self-supervised learning (SSL) to learn high-level speech representations has
been a popular approach to building Automatic Speech Recognition (ASR) systems
in low-resource settings. However, the common assumption made in literature is
that a considerable amount of unlabeled data is available for the same domain
or language that can be leveraged for SSL pre-training, which we acknowledge is
not feasible in a real-world setting. In this paper, as part of the Interspeech
Gram Vaani ASR challenge, we try to study the effect of domain, language,
dataset size, and other aspects of our upstream pre-training SSL data on the
final performance low-resource downstream ASR task. We also build on the
continued pre-training paradigm to study the effect of prior knowledge
possessed by models trained using SSL. Extensive experiments and studies reveal
that the performance of ASR systems is susceptible to the data used for SSL
pre-training. Their performance improves with an increase in similarity and
volume of pre-training data. We believe our work will be helpful to the speech
community in building better ASR systems in low-resource settings and steer
research towards improving generalization in SSL-based pre-training for speech
systems.
