  Recent advances in automatic speech recognition (ASR) often rely on large
speech foundation models for generating high-quality transcriptions. However,
these models can be impractical due to limited computing resources. The
situation is even more severe in terms of more realistic or difficult
scenarios, such as code-switching ASR (CS-ASR). To address this, we present a
framework for developing more efficient models for CS-ASR through knowledge
distillation using realistic speech-only data. Our proposed method, Leave No
Knowledge Behind During Knowledge Distillation (K$^2$D), leverages both the
teacher model's knowledge and additional insights from a small auxiliary model.
We evaluate our approach on two in-domain and two out-domain datasets,
demonstrating that K$^2$D is effective. By conducting K$^2$D on the unlabeled
realistic data, we have successfully obtained a 2-time smaller model with
5-time faster generation speed while outperforming the baseline methods and the
teacher model on all the testing sets. We have made our model publicly
available on Hugging Face
(https://huggingface.co/andybi7676/k2d-whisper.zh-en).
