  Self-supervised learning (SSL) methods have proven to be very successful in
automatic speech recognition (ASR). These great improvements have been reported
mostly based on highly curated datasets such as LibriSpeech for non-streaming
End-to-End ASR models. However, the pivotal characteristics of SSL is to be
utilized for any untranscribed audio data. In this paper, we provide a full
exploration on how to utilize uncurated audio data in SSL from data
pre-processing to deploying an streaming hybrid ASR model. More specifically,
we present (1) the effect of Audio Event Detection (AED) model in data
pre-processing pipeline (2) analysis on choosing optimizer and learning rate
scheduling (3) comparison of recently developed contrastive losses, (4)
comparison of various pre-training strategies such as utilization of in-domain
versus out-domain pre-training data, monolingual versus multilingual
pre-training data, multi-head multilingual SSL versus single-head multilingual
SSL and supervised pre-training versus SSL. The experimental results show that
SSL pre-training with in-domain uncurated data can achieve better performance
in comparison to all the alternative out-domain pre-training strategies.
