  Recent work on distilling Whisper's knowledge into small models using
pseudo-labels shows promising performance while reducing the size by up to
50\%. This results in small, efficient, and dedicated models. However, a
critical step of distillation from pseudo-labels involves filtering
high-quality predictions and using only those during training. This step
requires ground truth to compare and filter bad examples making the whole
process supervised. In addition to that, the distillation process requires a
large amount of data thereby limiting the ability to distil models in
low-resource settings. To address this challenge, we propose an unsupervised or
label-free framework for distillation, thus eliminating the requirement for
labeled data altogether. Through experimentation, we show that our
best-distilled models outperform the teacher model by 5-7 points in terms of
WER. Additionally, our models are on par with or better than similar supervised
data filtering setup. When we scale the data, our models significantly
outperform all zero-shot and supervised models. We demonstrate that it is
possible to distill large Whisper models into relatively small models without
using any labeled data. Our distilled models are 25-50\% more compute and
memory efficient while maintaining performance equal to or better than the
teacher model.
