  Transformer-based models have recently become very popular for
sequence-to-sequence applications such as machine translation and speech
recognition. This work proposes a dual-decoder transformer model for
low-resource multilingual speech recognition for Indian languages. Our proposed
model consists of a Conformer [1] encoder, two parallel transformer decoders,
and a language classifier. We use a phoneme decoder (PHN-DEC) for the phoneme
recognition task and a grapheme decoder (GRP-DEC) to predict grapheme sequence
along with language information. We consider phoneme recognition and language
identification as auxiliary tasks in the multi-task learning framework. We
jointly optimize the network for phoneme recognition, grapheme recognition, and
language identification tasks with Joint CTC-Attention [2] training. Our
experiments show that we can obtain a significant reduction in WER over the
baseline approaches. We also show that our dual-decoder approach obtains
significant improvement over the single decoder approach.
