  Building a multilingual Automated Speech Recognition (ASR) system in a
linguistically diverse country like India can be a challenging task due to the
differences in scripts and the limited availability of speech data. This
problem can be solved by exploiting the fact that many of these languages are
phonetically similar. These languages can be converted into a Common Label Set
(CLS) by mapping similar sounds to common labels. In this paper, new approaches
are explored and compared to improve the performance of CLS based multilingual
ASR model. Specific language information is infused in the ASR model by giving
Language ID or using CLS to Native script converter on top of the CLS
Multilingual model. These methods give a significant improvement in Word Error
Rate (WER) compared to the CLS baseline. These methods are further tried on
out-of-distribution data to check their robustness.
