  Self-supervised learning (SSL) has been able to leverage unlabeled data to
boost the performance of automatic speech recognition (ASR) models when we have
access to only a small amount of transcribed speech data. However, this raises
the question of which subset of the available unlabeled data should be selected
for transcription. Our work investigates different unsupervised data selection
techniques for fine-tuning the HuBERT model under a limited transcription
budget. We investigate the impact of speaker diversity, gender bias, and topic
diversity on the downstream ASR performance. We also devise two novel
techniques for unsupervised data selection: pre-training loss based data
selection and the perplexity of byte pair encoded clustered units (PBPE) and we
show how these techniques compare to pure random data selection. Finally, we
analyze the correlations between the inherent characteristics of the selected
fine-tuning subsets as well as how these characteristics correlate with the
resultant word error rate. We demonstrate the importance of token diversity,
speaker diversity, and topic diversity in achieving the best performance in
terms of WER.
