  The end-to-end ASR model is often desired in the streaming multilingual
scenario since it is easier to deploy and can benefit from pre-trained speech
models such as powerful foundation models. Meanwhile, the heterogeneous nature
and imbalanced data abundance of different languages may cause performance
degradation, leading to asynchronous peak performance for different languages
during training, especially on tail ones. Sometimes even the data itself may
become unavailable as a result of the enhanced privacy protection. Existing
work tend to significantly increase the model size or learn language-specific
decoders to accommodate each language separately. In this study, we explore
simple yet effective Language-Dependent Adapter (LDA) finetuning under a
cascaded Conformer transducer framework enhanced by teacher pseudo-labeling for
tail languages in the streaming multilingual ASR. The adapter only accounts for
0.4% of the full model per language. It is plugged into the frozen foundation
model and is the only trainable module during the finetuning process with noisy
student training. The final model merges the adapter parameters from different
checkpoints for different languages. The model performance is validated on a
challenging multilingual dictation dataset, which includes 39 tail languages
across Latin, Greek, Arabic, etc. Our proposed method brings 12.2% word error
rate reduction on average and up to 37.5% on a single locale. Furthermore, we
show that our parameter-efficient LDA can match the quality of the full model
finetuning, thus greatly alleviating the asynchronous peak performance issue.
