  Whisper is a multitask and multilingual speech model covering 99 languages.
It yields commendable automatic speech recognition (ASR) results in a subset of
its covered languages, but the model still underperforms on a non-negligible
number of under-represented languages, a problem exacerbated in smaller model
versions. In this work, we examine its limitations, demonstrating the presence
of speaker-related (gender, age) and model-related (resourcefulness and model
size) bias. Despite that, we show that only model-related bias are amplified by
quantization, impacting more low-resource languages and smaller models.
Searching for a better compression approach, we propose DistilWhisper, an
approach that is able to bridge the performance gap in ASR for these languages
while retaining the advantages of multitask and multilingual capabilities. Our
approach involves two key strategies: lightweight modular ASR fine-tuning of
whisper-small using language-specific experts, and knowledge distillation from
whisper-large-v2. This dual approach allows us to effectively boost ASR
performance while keeping the robustness inherited from the multitask and
multilingual pre-training. Results demonstrate that our approach is more
effective than standard fine-tuning or LoRA adapters, boosting performance in
the targeted languages for both in- and out-of-domain test sets, while
introducing only a negligible parameter overhead at inference.
