  Semi-supervised learning (SSL) is an active area of research which aims to
utilize unlabelled data in order to improve the accuracy of speech recognition
systems. The current study proposes a methodology for integration of two key
ideas: 1) SSL using connectionist temporal classification (CTC) objective and
teacher-student based learning 2) Designing effective data-selection mechanisms
for leveraging unlabelled data to boost performance of student models. Our aim
is to establish the importance of good criteria in selecting samples from a
large pool of unlabelled data based on attributes like confidence measure,
speaker and content variability. The question we try to answer is: Is it
possible to design a data selection mechanism which reduces dependence on a
large set of randomly selected unlabelled samples without compromising on Word
Error Rate (WER)? We perform empirical investigations of different data
selection methods to answer this question and quantify the effect of different
sampling strategies. On a semi-supervised ASR setting with 40000 hours of
carefully selected unlabelled data, our CTC-SSL approach gives 17% relative WER
improvement over a baseline CTC system trained with labelled data. It also
achieves on-par performance with CTC-SSL system trained on order of magnitude
larger unlabeled data based on random sampling.
