  Even with several advancements in multilingual modeling, it is challenging to
recognize multiple languages using a single neural model, without knowing the
input language and most multilingual models assume the availability of the
input language. In this work, we propose a novel bilingual end-to-end (E2E)
modeling approach, where a single neural model can recognize both languages and
also support switching between the languages, without any language input from
the user. The proposed model has shared encoder and prediction networks, with
language-specific joint networks that are combined via a self-attention
mechanism. As the language-specific posteriors are combined, it produces a
single posterior probability over all the output symbols, enabling a single
beam search decoding and also allowing dynamic switching between the languages.
The proposed approach outperforms the conventional bilingual baseline with
13.3%, 8.23% and 1.3% word error rate relative reduction on Hindi, English and
code-mixed test sets, respectively.
